A guide on configuration file

Manticore search uses a configuration file for customizing settings and declaration indexes and sources.

The configuration file is in a plain text format and can be edited with any text editor. The configuration is logically split into sections. It’s content is enclosed in { and }. There are 5 types of sections as follow:

  • searchd - mandatory and can be declared only once, contains settings of searchd daemon
  • indexer - optional, can be declared only once, contains settings for indexer tool
  • common - optional, can be declared only once, contains common settings for searchd and indexer
  • index - can be declared multiple times, supports inheritance, requires declaration of an index name, contains configuration of an index. At least one index must be declared.
  • source - can be declared multiple times, supports inheritance, requires declaration of a source name, contains configuration of a source, optional

Indexes and sources are parsed as the configuration file is read. In case of inherited sections, child sections must come after parent declarations. The same applies to distributed indexes with local indexes.

There is no rule for the settings sections. They can declared at the start or end of the file or even mixed between index/source declarations.

Special suffixes

Manticore search recognizes and parses special suffixes which makes easier to use numeric values with special meaning. Common form for them is integer number + literal, like 10k or 100d, but not 40.3s (since 40.3 is not integer), or not 2d 4h (since there are two, not one value). Literals are case-insensitive, so 10W is the same as 10w. There are 2 types of such suffixes currently supported:

  • Size suffixes - can be used in parameters that define size of something (memory buffer, disk file, limit of RAM, etc. ) in bytes. “Naked” numbers in that places mean literally size in bytes (octets). Size values take suffix k for kilobytes (1k=1024), m for megabytes (1m=1024k), g for gigabytes (1g=1024m) and t for terabytes (1t=1024g).
  • Time suffixes - can be used in parameters defining some time interval values like delays, timeouts, etc. “Naked” values for those parameters usually have documented scale, and you must know if their numbers, say, 100, means ‘100 seconds’ or ‘100 milliseconds’. However instead of guessing you just can write suffixed value and it will be fully determined by it’s suffix. Time values take suffix us for useconds (microseconds), ms for milliseconds, s for seconds, m for minutes, h for hours, d for days and w for weeks.

Warning

Giga-, and especially tera- size suffixes are not very usable right now, since most of the sizes inside are limited by 2Gb (or, being precise, 2Gb - 1 byte), and for the moment only rt_mem_limit, attr_update_reserve from index config, and qcache_max_bytes from searchd config accepts 64-bit values which may exceed 2Gb.

Scripting support

The configuration support shebang syntax - this means the configuration can be written in a programming language and interpreted at loading, allowing dynamic settings.

For example, indexes can be generated by querying a database table, various settings can be modified depending on external factors or external files can be included (which contain indexes and/sources).

The configuration file is parsed by declared declared interpreter and the output is used as the actual configuration. This is happening each time the configuration is read (not only at searchd startup).

This facility is not available on Windows platform.

In the following example, we are using php to create multiple indexes with different name and we also scan a specific folder for file containing extra declarations of indexes.

#!/usr/bin/php
...
<?php for ($i=1; $i<=6; $i++) { ?>
index test_<?=$i?> {
  type = rt
  path = /var/lib/manticore/data/test_<?=$i?>
  rt_field = subject
  ...
 }
 <?php } ?>
 ...

 <?php
 $confd_folder='/etc/manticore.conf.d/';
 $files = scandir($confd_folder);
 foreach($files as $file)
 {
         if(($file == '.') || ($file =='..'))
         {}else{
                 $fp = new SplFileInfo($confd_folder.$file);
                 if('conf' == $fp->getExtension()){
                         include ($confd_folder.$file);
                 }
         }
 }
 ?>

Comments

The configuration file supports comments, with # character used as start comment section. The comment character can be present at the start of the line or inline.

Extra care should be considered when using # in character tokenization settings as everything after it will not be taken into consideration. To avoid this, use # UTF-8 which is U+23.

# can also be escaped using \. Escaping is required if # is present in database credential in source declarations.

Inheritance of index and source declarations

Both index and source declarations support inheritance. This allows a better organization of indexes having similar settings or structure and reduce the size of the configuration.

For a parent index/source nothing needs to be specified.

For the child index/source the declaration will contain the index/source name followed by : and the parent name.

index parent {
path = /var/lib/manticore/parent
...
}

index child:parent {
path = /var/lib/manticore/child
...
}

The child will inherit the entire configuration of the parent. In the child declaration any setting declared will overwrite the inherited values. Please note that in case of multi value settings, defining a single value in child will clear out all inherited values. For example in the parent there are several sql_query_pre declaration and the child has a single sql_query_pre declaration, all the sql_query_pre inherited declarations are cleared. If you need to override some of the inherited values from parent, they need to be explicit declared in the child. This is also available if you don’t need a value from parent. For example if the value of sql_query_pre from parent is not needed, then in the child we will declare the directive with an empty value like sql_query_pre=. This also means that existing values of a multi value setting will not be copied if the child declares one value for that setting. The inheritance bevahiour applies to fields and attributes and not just index options. If, for example, the parent has 2 integer attributes and the child needs a new integer attribute, the integer attributes declaration from parent must be copied in the child configuration.