# Welcome to Manticore Search’s Documentation¶

## Introduction¶

Manticore Search is a solution for building high-performance search applications. It is a fork of Sphinx search engine and, as it’s predecesor, it’s written in C++ and distributed under open source license GPL version 2.

While most modern databases include full-text indexes, they have two issues:

• they are not fast enough. While some can perform well on simple searches, their performance get hit when doing complex text searches or when adding other filtering
• the full-text capabilities are limited, which can be the lack of ability to customize the tokenization process or lack of operators that can perform advanced matching

hence the need for a dedicated full-text search engine, which can be fast but at the same time offer advanced features.

In most cases, it is used to assist a traditional database by replacing the full-text search operations and in some cases it can help on non-full-text operations as well. A special utility is shipped along with the main component which can grab data from various sources (databases, XML or TSV/CSV pipelines) and build the indexes.

Manticore Search is similar in usage to a database: an application connects to it, performs queries, gets results, returns them to end user.

Typical flow with Manticore and a database

Data can be also be fed into indexes like in a traditional database, in which case indexing is near real-time, without any dependency or connection to a specific database. Regardless of how data is indexed, there is no difference at searching.

Indexes are similar to a data table, with rows called and viewed as documents and columns as fields and attributes. JSON attribute/column is available for schemaless data, however full-text matching is not yet available on it. Manticore Search offers several ways of connecting:

• an own implementation of the MySQL network protocol, using a small SQL subset called SphinxQL. Any MySQL client/library can be used. Currently this is the recommended interface.
• an HTTP API , which can accept search queries in SQL, native or JSON format
• a native API, with official and 3rd party clients available for many languages
• a proxy storage engine for MySQL called SphinxSE
• an interface for MySQL FEDERATED storage engine

In addition to regular text searching, Manticore can also perform inverse search (percolate queries), hit highlighting, word corrections, attribute faceting, geospatial searches. Manticore is designed to scale by supporting distributing a data collection across multiple servers and offer an internal load-balancer for High Availability.

## Gettting Started¶

### Getting started using Docker container¶

#### Installing and running¶

$docker run --name manticore -p 9306:9306 -d manticoresearch/manticore  The Manticore Search container doesn’t have a persistent storage and in case the container is stopped, any changes are lost. For persistence, there are 3 folders that can be mounted locally: • /etc/sphinxsearch - location of sphinx.conf • /var/lib/manticore/data - used for index files • /var/lib/manticore/log - used for log files The run command becomes: $ docker run --name manticore -v ~/manticore/etc/:/etc/sphinxsearch/ -v ~/manticore/data/:/var/lib/manticore/data -v ~/manticore/logs/:/var/lib/manticore/log -p 9306:9306 -d manticoresearch/manticore


In ~/manticore/ you need to create the etc/ , data/ and logs/ folders, as well as add a valid sphinx.conf in ~/manticore/etc/.

#### Running queries¶

The simple way to connect and do some tests is to use the SphinxQL protocol. For this, you need a mysql command line client.

While it implements the MySQL protocol, SphinxQL is not 100% compatible with MySQL syntax. There are specific extensions, like MATCH clause [the most powerful thing in Manticore] or WITHIN GROUP BY and many functions available in MySQL are not implemented (or they are dummy just to allow compatibility with MySQL connector e.g.) or JOINs between indexes which are not supported yet.

First, let’s connect to Manticore Search and take a look at the available indexes:

$mysql -P9306 -h0  mysql> SHOW TABLES; +-------+-------------+ | Index | Type | +-------+-------------+ | testrt| rt | +-------+-------------+ 2 rows in set (0.00 sec)  Now let’s look at our RT index: mysql> DESCRIBE testrt; +---------+--------+ | Field | Type | +---------+--------+ | id | bigint | | title | field | | content | field | | gid | uint | +---------+--------+ 4 rows in set (0.00 sec)  As the RT indexes start empty, let’s add some data into it first mysql> INSERT INTO testrt VALUES(1,'List of HP business laptops','Elitebook Probook',10); Query OK, 1 row affected (0.00 sec) mysql> INSERT INTO testrt VALUES(2,'List of Dell business laptops','Latitude Precision Vostro',10); Query OK, 1 row affected (0.00 sec) mysql> INSERT INTO testrt VALUES(3,'List of Dell gaming laptops','Inspirion Alienware',20); Query OK, 1 row affected (0.00 sec) mysql> INSERT INTO testrt VALUES(4,'Lenovo laptops list','Yoga IdeaPad',30); Query OK, 1 row affected (0.01 sec) mysql> INSERT INTO testrt VALUES(5,'List of ASUS ultrabooks and laptops','Zenbook Vivobook',30); Query OK, 1 row affected (0.01 sec)  Now we have some data, we can do some queries. Fulltext searches are done with the special clause MATCH, which is the main workhorse. mysql> SELECT * FROM testrt WHERE MATCH('list of laptops'); +------+------+ | id | gid | +------+------+ | 1 | 10 | | 2 | 10 | | 3 | 20 | | 5 | 30 | +------+------+ 4 rows in set (0.00 sec)  As you see in the result set we can only get back the doc id and the attributes. The fulltext fields values are not returned since the text is only indexed, not stored also, and it’s impossible to rebuild the original text. Now let’s add some filtering and more ordering: mysql> SELECT *,WEIGHT() FROM testrt WHERE MATCH('list of laptops') AND gid>10 ORDER BY WEIGHT() DESC,gid DESC; +------+------+----------+ | id | gid | weight() | +------+------+----------+ | 5 | 30 | 2334 | | 3 | 20 | 2334 | +------+------+----------+ 2 rows in set (0.00 sec)  The WEIGHT() function returns the calculated matching score. If no ordering specified, the result is sorted descending by the score provided by WEIGHT(). In this example we order first by weight and then by an integer attribute. The search above does a simple matching, where all words need to be present. But we can do more (and this is just a simple example): mysql> SELECT *,WEIGHT() FROM testrt WHERE MATCH('"list of business laptops"/3'); +------+------+----------+ | id | gid | weight() | +------+------+----------+ | 1 | 10 | 2397 | | 2 | 10 | 2397 | | 3 | 20 | 2375 | | 5 | 30 | 2375 | +------+------+----------+ 4 rows in set (0.00 sec) mysql> SHOW META; +---------------+----------+ | Variable_name | Value | +---------------+----------+ | total | 4 | | total_found | 4 | | time | 0.000 | | keyword[0] | list | | docs[0] | 5 | | hits[0] | 5 | | keyword[1] | of | | docs[1] | 4 | | hits[1] | 4 | | keyword[2] | business | | docs[2] | 2 | | hits[2] | 2 | | keyword[3] | laptops | | docs[3] | 5 | | hits[3] | 5 | +---------------+----------+ 15 rows in set (0.00 sec)  Here we search for 4 words, but we can have a match even if only 3 words (of 4) are found. The search will rank higher first the documents that contain all the words. We also added a SHOW META command. SHOW META returns information about previous executed query, that is number of found records (in total_found), execution time (in time) and statistics about the keywords of the search. To create a new RT index, you need to define it in the sphinx.conf. A simple definition looks like: index myindexname { type = rt path = /path/to/myrtindex rt_mem_limit = 256M rt_field = title rt_attr_uint = attr1 rt_attr_uint = attr2 }  To get the index online you need to either restart the daemon or send a HUP signal to it. #### Using plain indexes¶ Unlike RT, a plain also requires configuring a source for it. In our example we are using a MySQL source. Add in your sphinx.conf: source src1 { type = mysql sql_host = 172.17.0.1 sql_user = test sql_pass = sql_db = test sql_port = 3306 # optional, default is 3306 sql_query_pre = SET NAMES utf8 sql_query = \ SELECT id, group_id, UNIX_TIMESTAMP(date_added) AS date_added, title, content \ FROM documents sql_attr_uint = group_id sql_attr_timestamp = date_added } index test1 { source = src1 path = /var/lib/manticore/data/test1 min_word_len = 1 }  In this example we assume we have a MySQL instance on the local host, but as Manticore Search runs inside a Docker container, we need to use ‘172.17.0.1’, the static IP address of the Docker host. For more details, please check Docker documentation. You also need to adjust the MySQL credentials accordingly. Then we look after the sql_query, which is the query that grabs the data sql_query = \ SELECT id, group_id, UNIX_TIMESTAMP(date_added) AS date_added, title, content \ FROM documents  For a quick test, we’re going to use the following sample table in MySQL: DROP TABLE IF EXISTS test.documents; CREATE TABLE test.documents ( id INTEGER PRIMARY KEY NOT NULL AUTO_INCREMENT, group_id INTEGER NOT NULL, date_added DATETIME NOT NULL, title VARCHAR(255) NOT NULL, content TEXT NOT NULL ); INSERT INTO test.documents ( id, group_id, date_added, title, content ) VALUES ( 1, 1, NOW(), 'test one', 'this is my test document number one. also checking search within phrases.' ), ( 2, 1, NOW(), 'test two', 'this is my test document number two' ), ( 3, 2, NOW(), 'another doc', 'this is another group' ), ( 4, 2, NOW(), 'doc number four', 'this is to test groups' );  If you want to use another table, keep in mind that the first column in the result set must be an unsigned unique integer - for most cases this is your primary key id of a table. If not specified, the rest of the columns are indexed as fulltext fields. Columns which should be used as attributes need to be declared. In our example group_id and date_added are attributes: sql_attr_uint = group_id sql_attr_timestamp = date_added  Once we have this setup, we can run the indexing process: $ docker exec -it manticore indexer  test1  --rotate
using config file '/etc/sphinxsearch/sphinx.conf'...
indexing index 'test1'...
collected 4 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 4 docs, 193 bytes
total 0.015 sec, 12335 bytes/sec, 255.65 docs/sec
total 4 reads, 0.000 sec, 8.1 kb/call avg, 0.0 msec/call avg
total 12 writes, 0.000 sec, 0.1 kb/call avg, 0.0 msec/call avg


Index is created and is ready to be used:

mysql> SHOW TABLES;
+-------+-------------+
| Index | Type        |
+-------+-------------+
| dist1 | distributed |
| rt    | rt          |
| test1 | local       |
+-------+-------------+
3 rows in set (0.00 sec)

mysql> SELECT * FROM test1;
+------+----------+------------+
| id   | group_id | date_added |
+------+----------+------------+
|    1 |        1 | 1507904567 |
|    2 |        1 | 1507904567 |
|    3 |        2 | 1507904567 |
|    4 |        2 | 1507904567 |
+------+----------+------------+
4 rows in set (0.00 sec)


A quick test of a search which should match 2 terms, but not match another one:

mysql> SELECT * FROM test1 WHERE MATCH('test document -one');
+------+----------+------------+-------+
| id   | group_id | date_added | tag   |
+------+----------+------------+-------+
|    2 |        1 | 1519040667 | 2,4,6 |
+------+----------+------------+-------+
1 row in set (0.00 sec)


### Getting Started using official packages¶

#### Installing and running¶

$wget https://github.com/manticoresoftware/manticore/releases/download/x.y.z/manticore_z.y.z.deb$ sudo dpkg -i manticore_x.y.z.deb


Start the service:

$systemctl start manticore  or $ service manticore start


depending on the distribution used.

At this point you can start using Manticore Search. The configuration file is located at /etc/sphinx/sphinx.conf (RHEL/CentOS) or /etc/sphinxsearch/sphinx.conf (Debian/Ubuntu). The default configuration comes with an empty RT index ready to be used and a sample plain index and interfaces SphinxQL on port 9306 and native API on port 9312.

You can also compile Manticore Search from sources. Compilation is easy and uses cmake and you can also create packages for your operating system.

#### Running queries¶

The simple way to connect and do some tests is to use the SphinxQL protocol. For this, you need a mysql command line client.

While it implements the MySQL protocol, SphinxQL is not 100% compatible with MySQL syntax. There are specific extensions, like MATCH clause [the most powerful thing in Manticore] or WITHIN GROUP BY and many functions available in MySQL are not implemented (or they are dummy just to allow compatibility with MySQL connector e.g.) or JOINs between indexes which are not supported yet.

First, let’s connect to Manticore Search and take a look at the available indexes:

$mysql -P9306 -h0  mysql> SHOW TABLES; +-------+-------------+ | Index | Type | +-------+-------------+ | dist1 | distributed | | testrt| rt | +-------+-------------+ 2 rows in set (0.00 sec)  Now let’s look at our RT index: mysql> DESCRIBE testrt; +---------+--------+ | Field | Type | +---------+--------+ | id | bigint | | title | field | | content | field | | gid | uint | +---------+--------+ 4 rows in set (0.00 sec)  As the RT indexes start empty, let’s add some data into it first mysql> INSERT INTO testrt VALUES(1,'List of HP business laptops','Elitebook Probook',10); Query OK, 1 row affected (0.00 sec) mysql> INSERT INTO testrt VALUES(2,'List of Dell business laptops','Latitude Precision Vostro',10); Query OK, 1 row affected (0.00 sec) mysql> INSERT INTO testrt VALUES(3,'List of Dell gaming laptops','Inspirion Alienware',20); Query OK, 1 row affected (0.00 sec) mysql> INSERT INTO testrt VALUES(4,'Lenovo laptops list','Yoga IdeaPad',30); Query OK, 1 row affected (0.01 sec) mysql> INSERT INTO testrt VALUES(5,'List of ASUS ultrabooks and laptops','Zenbook Vivobook',30); Query OK, 1 row affected (0.01 sec)  Now we have some data, we can do some queries. Fulltext searches are done with the special clause MATCH, which is the main workhorse. mysql> SELECT * FROM testrt WHERE MATCH('list of laptops'); +------+------+ | id | gid | +------+------+ | 1 | 10 | | 2 | 10 | | 3 | 20 | | 5 | 30 | +------+------+ 4 rows in set (0.00 sec)  As you see in the result set we can only get back the doc id and the attributes. The fulltext fields values are not returned since the text is only indexed, not stored also, and it’s impossible to rebuild the original text. Now let’s add some filtering and more ordering: mysql> SELECT *,WEIGHT() FROM testrt WHERE MATCH('list of laptops') AND gid>10 ORDER BY WEIGHT() DESC,gid DESC; +------+------+----------+ | id | gid | weight() | +------+------+----------+ | 5 | 30 | 2334 | | 3 | 20 | 2334 | +------+------+----------+ 2 rows in set (0.00 sec)  The WEIGHT() function returns the calculated matching score. If no ordering specified, the result is sorted descending by the score provided by WEIGHT(). In this example we order first by weight and then by an integer attribute. The search above does a simple matching, where all words need to be present. But we can do more (and this is just a simple example): mysql> SELECT *,WEIGHT() FROM testrt WHERE MATCH('"list of business laptops"/3'); +------+------+----------+ | id | gid | weight() | +------+------+----------+ | 1 | 10 | 2397 | | 2 | 10 | 2397 | | 3 | 20 | 2375 | | 5 | 30 | 2375 | +------+------+----------+ 4 rows in set (0.00 sec) mysql> SHOW META; +---------------+----------+ | Variable_name | Value | +---------------+----------+ | total | 4 | | total_found | 4 | | time | 0.000 | | keyword[0] | list | | docs[0] | 5 | | hits[0] | 5 | | keyword[1] | of | | docs[1] | 4 | | hits[1] | 4 | | keyword[2] | business | | docs[2] | 2 | | hits[2] | 2 | | keyword[3] | laptops | | docs[3] | 5 | | hits[3] | 5 | +---------------+----------+ 15 rows in set (0.00 sec)  Here we search for 4 words, but we can have a match even if only 3 words (of 4) are found. The search will rank higher first the documents that contain all the words. We also added a SHOW META command. SHOW META returns information about previous executed query, that is number of found records (in total_found), execution time (in time) and statistics about the keywords of the search. To create a new RT index, you need to define it in the sphinx.conf. A simple definition looks like: index myindexname { type = rt path = /path/to/myrtindex rt_mem_limit = 256M rt_field = title rt_attr_uint = attr1 rt_attr_uint = attr2 }  To get the index online you need to either restart the daemon or send a HUP signal to it. #### Using plain indexes¶ Unlike RT, the plain index requires setting up the source and run the indexing process which gathers the data. For this we need to edit the sphinx.conf configuration file. The initial configuration comes with a sample plain index along with a source. For simplicity we use a MySQL source. First, the database credentials need to be adjusted ... sql_host = localhost sql_user = test sql_pass = sql_db = test sql_port = 3306 # optional, default is 3306 ...  Then we look after the sql_query, which is the query that grabs the data sql_query = \ SELECT id, group_id, UNIX_TIMESTAMP(date_added) AS date_added, title, content \ FROM documents  For a quick test, we’re going to use the following sample table in MySQL: DROP TABLE IF EXISTS test.documents; CREATE TABLE test.documents ( id INTEGER PRIMARY KEY NOT NULL AUTO_INCREMENT, group_id INTEGER NOT NULL, date_added DATETIME NOT NULL, title VARCHAR(255) NOT NULL, content TEXT NOT NULL ); INSERT INTO test.documents ( id, group_id, date_added, title, content ) VALUES ( 1, 1, NOW(), 'test one', 'this is my test document number one. also checking search within phrases.' ), ( 2, 1, NOW(), 'test two', 'this is my test document number two' ), ( 3, 2, NOW(), 'another doc', 'this is another group' ), ( 4, 2, NOW(), 'doc number four', 'this is to test groups' );  If you want to use your table, you need make some changes in the source definition. One is to modify the sql_query. Keep in mind that the first column in the result set must be an unsigned unique integer - for most cases this is your primary key id of a table. If not specified, the rest of the columns are indexed as fulltext fields. Columns which should be used as attributes need to be declared. In our example group_id and date_added are attributes: sql_attr_uint = group_id sql_attr_timestamp = date_added  Once we have this setup, we can run the indexing process: $ sudo -u manticore  indexer test1  --rotate
using config file '/etc/sphinxsearch/sphinx.conf'...
indexing index 'test1'...
collected 4 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 4 docs, 193 bytes
total 0.015 sec, 12335 bytes/sec, 255.65 docs/sec
total 4 reads, 0.000 sec, 8.1 kb/call avg, 0.0 msec/call avg
total 12 writes, 0.000 sec, 0.1 kb/call avg, 0.0 msec/call avg


Index is created and is ready to be used:

mysql> SHOW TABLES;
+-------+-------------+
| Index | Type        |
+-------+-------------+
| dist1 | distributed |
| testrt| rt          |
| test1 | local       |
+-------+-------------+
3 rows in set (0.00 sec)

mysql> SELECT * FROM test1;
+------+----------+------------+
| id   | group_id | date_added |
+------+----------+------------+
|    1 |        1 | 1507904567 |
|    2 |        1 | 1507904567 |
|    3 |        2 | 1507904567 |
|    4 |        2 | 1507904567 |
+------+----------+------------+
4 rows in set (0.00 sec)


A quick test of a search which should match 2 terms, but not match another one:

mysql> SELECT * FROM test1 WHERE MATCH('test document -one');
+------+----------+------------+-------+
| id   | group_id | date_added | tag   |
+------+----------+------------+-------+
|    2 |        1 | 1519040667 | 2,4,6 |
+------+----------+------------+-------+
1 row in set (0.00 sec)


### Migrating from Manticore or Sphinx Search 2.x¶

Upgrading from 2.x to 3.x is not straightforward, because the index storage engine received a massive upgrade and the new searchd can’t load older indexes and upgrade them to new format on-the-fly.

The upgrade procedure may differ depending on your setup (number of servers in the cluster, whether you have HA or not etc.), but in general it’s about creating new 3.x index versions and replacing your existing ones with them along with replacing older 2.x binaries with the new ones.

There are two special requirements to take care:

• RealTime indexes require to be flushed on existing version (see below)
• Plain indexes with kill-lists require adding a new directive in index configuration (see below)

Manticore Search 3 includes a new tool - index_converter tool - that can convert 2.x indexes to 3.x format. The index_converter tool comes in a separate package which should be installed first. Using the convert tool create 3.x versions of your indexes. The index_converter can write the new files in the existing data folder and backup the old files or it can write the new files to a chosen folder. If your

If you have a single server:

• install manticore-converter package
• use index_converter to create new versions of the indexes in a different folder than the existing data folder ( using –output-dir option)
• stop existing Manticore, upgrade to 3.0, move the new indexes to data folder, start Manticore

To get a minimal downtime, you can copy 2.x indexes, config (you’ll need to edit paths here for indexes, logs and different ports) and binaries to a separate location and start this on a separate port and point your application to it.After upgrade is made to  3.0 and the new daemon is started, you can point back the application to the normal ports. If all good, stop the 2.x copy and delete the files to free the space.

If you have a spare box (like a testing or staging server), you can do here first the index upgrade and even install Manticore 3 to perform several tests and if everything is ok copy the new index files to the production server.If you have multiple servers which can be pulled out from production, do it one by one and perform the upgrade on each. For distributed setups, 2.x searchd can work as master with 3.x nodes, so you can do  upgrading on the data nodes first and at the end the master node.

There have been no changes made on how clients should connect to the engine or any change in querying mode or queries behavior.

#### Index storage engine¶

Manticore Search 3 got a redesigned index storage. Indexes created with Manticore/Sphinx 2.x cannot be loaded by Manticore Search 3.

Existing indexes (RealTime or plain) can be converted with index_converter tool or rebuilt from scratch. In case of RealTime indexes before converting one you need to flush it’s memory chunk to disk using FLUSH RAMCHUNK.

Because of the 4GB limitation, a RealTime index in 2.x could still have several disk chunks after an optimize operation. After upgrading to 3.x, these indexes can now be optimized to 1-disk chunk with the usual OPTIMIZE command.

Index files also changed. The only component that didn’t get any structural changes is the spp file (hitlists). sps (strings/json) and spm (MVA) are now held by spb (var-length attributes). The new format has an spm file present, but it’s used for row map (previously it was dedicated for MVA attributes). The new extensions added are spt (docid lookup), sphi ( secondary index histograms). In case you are using scripts that manipulate index files, they should be adapted for the new file extensions.

#### Kill-list¶

Kill-lists have been redesigned in Manticore Search 3. In previous versions kill-lists were applied on the result set provided by each previous searched index on query time.

Thus In 2.x the index order at query time mattered. For example if a delta index had a kill-list in order to apply it against the main index the order had to be main, delta (either in a distributed index or in the FROM clause).

In Manticore 3 kill-lists are applied to an index when it’s loaded during searchd startup or gets rotated. New directive killlist_target in index configuration specifies target indexes and defines which doc ids from the source index should be used for suppression. These can be ids from the defined kill-list, actual doc ids of the index or the both.

Documents from the kill-lists are deleted from the target indexes, they are not returned in results even if the search doesn’t include the index that provided the kill-lists. Because of that the order of indexes for searching does not matter any more. Now delta,main and main,delta will provide the same results.

In previous versions indexes were rotated following the order from the configuration file. In Manticore 3 index rotation order is much smarter and works in accordance with killlist targets. Before starting to rotate indexes the daemon looks for chains of indexes by killlist_target definitions. It will then first rotate indexes not referenced anywhere as kill-lists targets. Next it will rotate indexes targeted by already rotated indexes and so on. For example if we do indexer --all and we have 3 indexes : main, delta_big (which targets at the main) and delta_small (with target at delta_big), first the delta_small is rotated, then delta_big and finally the main. This is to ensure that when a dependent index is rotated it gets the most actual kill-list from other indexes.

#### Removed configuration keys¶

• docinfo - everything is now extern
• inplace_docinfo_gap - not needed anymore
• mva_updates_pool - MVAs don’t have anymore a dedicated pool for updates, as now they can be updated directly in the blob (see below).

#### Updating var-length attributes¶

String, JSON and MVA attributes can now be updated using UPDATE statement.

In 2.x string attributes required REPLACE, for JSON it was only possible to update scalar properties (as they were fixed-width) and MVAs could be updated using the MVA pool. Now updates are performed directly on the blob component. One setting that may require tuning is attr_update_reserve which allows changing the allocated extra space at the end of the blob used to avoid frequent resizes in case the new values are bigger than the existing values in the blob.

#### Document IDs¶

Doc ids used to be UNSIGNED 64-bit integers. Now they are SIGNED 64-bit integers.

### A guide on configuration file¶

Manticore search uses a configuration file for customizing settings and declaration indexes and sources.

The configuration file is in a plain text format and can be edited with any text editor. The configuration is logically split into sections. It’s content is delimited by { and }. There are 5 types of sections as follow:

• searchd - mandatory and can be declared only once, contains settings of searchd daemon
• indexer - optional, can be declared only once, contains settings for indexer tool
• common - optional, can be declared only once, contains common settings for searchd and indexer
• index - can be declared multiple times, supports inheritance, requires declaration of an index name, contains configuration of an index. At least one index must be declared.
• source - can be declared multiple times, supports inheritance, requires declaration of a source name, contains configuration of a source, optional

Indexes and sources are parsed as the configuration file is read. In case of inherited sections, child sections must come after parent declarations. The same applies to distributed indexes with local indexes.

There is no rule for the settings sections. They can declared at the start or end of the file or even mixed between index/source declarations.

#### Scripting support¶

The configuration support shebang syntax - this means the configuration can be written in a programming language and interpreted at loading, allowing dynamic settings.

For example, indexes can be generated by querying a database table, various settings can be modified depending on external factors or external files can be included (which contain indexes and/sources).

The configuration file is parsed by declared declared interpreter and the output is used as the actual configuration. This is happening each time the configuration is read (not only at searchd startup).

This facility is not available on Windows platform.

In the following example, we are using php to create multiple indexes with different name and we also scan a specific folder for file containing extra declarations of indexes.

#!/usr/bin/php
...
<?php for ($i=1;$i<=6; $i++) { ?> index test_<?=$i?> {
type = rt
path = /var/lib/manticore/data/test_<?=$i?> rt_field = subject ... } <?php } ?> ... <?php$confd_folder='/etc/manticore.conf.d/';
$files = scandir($confd_folder);
foreach($files as$file)
{
if(($file == '.') || ($file =='..'))
{}else{
$fp = new SplFileInfo($confd_folder.$file); if('conf' ==$fp->getExtension()){
include ($confd_folder.$file);
}
}
}
?>


The configuration file supports comments, with # character used as start comment section. The comment character can be present at the start of the line or inline.

Extra care should be considered when using # in character tokenization settings as everything after it will not be taken into consideration. To avoid this, use # UTF-8 which is U+23.

# can also be escaped using \. Escaping is required if # is present in database credential in source declarations.

#### Inheritance of index and source declarations¶

Both index and source declarations support inheritance. This allows a better organization of indexes having similar settings or structure and reduce the size of the configuration.

For a parent index/source nothing needs to be specified.

For the child index/source the declaration will contain the index/source name followed by : and the parent name.

index parent {
path = /var/lib/manticore/parent
...
}

index child:parent {
path = /var/lib/manticore/child
...
}


The child will inherit the entire configuration of the parent. In the child declaration any setting declared will overwrite the inherited values. Please note that in case of multi value settings, defining a single value in child will clear out all inherited values. For example in the parent there are several sql_query_pre declaration and the child has a single sql_query_pre declaration, all the sql_query_pre inherited declarations are cleared. If you need to override some of the inherited values from parent, they need to be explicit declared in the child. This is also available if you don’t need a value from parent. For example if the value of sql_query_pre from parent is not needed, then in the child we will declare the directive with an empty value like sql_query_pre=. This also means that existing values of a multi value setting will not be copied if the child declares one value for that setting. The inheritance bevahiour applies to fields and attributes and not just index options. If, for example, the parent has 2 integer attributes and the child needs a new integer attribute, the integer attributes declaration from parent must be copied in the child configuration.

### A guide on connectivity¶

Manticore Search offers 3 protocols that allows clients to connect in multiple ways.

Please note that currently none of the protocols implement any kind of authentification. Someone able to connect to your search instances could crawl your data, run heavy queries that can slow down servers or perform changes on the indexes. Your search servers should not be reachable by outside world. If the application that connects to Manticore live on the same server, the listen clauses should bound the protocols only to local interface.

In case search instances require to be reached from outside, connections should be restricted with the help of a firewall only the the known IPs authorized to connect. For SphinxQL, proxies like ProxySQL can add an authentification level. For HTTP protocol, a reserve proxy like Nginx can implement HTTP based authentifications.

#### SphinxAPI protocol¶

The SphinxAPI protocol is used for both connecting with clients as well as connecting multiple daemon instances for cluster setups. The default port (and assigned by IANA) for SphinxAPI protocol is 9312.

Clients can connect to Manticore over SphinxAPI protocol using API client libraries. Officially, we support API client libraries for:

• C
• PHP
• Java
• Ruby
• Go

The Go client has it’s own repository on github , while the rest of the clients can be found in /api directory.

For other languages, 3rd party APi clients exists. Older API client libraries, including versions for Sphinx Search, should work with newer daemons, however they will not be able to access new features.

SphinxAPI protocol is also used by SphinxSE MySQL plugin, which acts as a proxy engine that allows running Manticore queries from within a MySQL server. For more information about SphinxSE see MySQL storage engine (SphinxSE).

#### SphinxQL protocol¶

SphinxQL is an implementation of MySQL 4.1 protocol. Manticore doesn’t implement the full range of SQL commands or functions available in MySQL. There are custom clauses and functions, like the MATCH(), implemented in MySQL that are not found in MYSQL.

Manticore Search doesn’t support server-side prepared statements. Client-side prepared statements can be used with Manticore. It must be noted that Manticore implements the multi value (MVA) data type for which there is no equivalent in MySQL or libraries implementing prepared statements. In these cases, the MVA values will need to be crafted in the raw query.

Manticore Search does not have the concept of database like MySQL and no user access control is yet implemented. However some MySQL clients/connectors demand values set for user/password or database name. These can be set arbitrarily as Manticore will simply ignore the values.

The default port (and assigned by IANA) for SphinxQL protocol is 9306.

##### MySQL Connectors¶

The official MySQL connectors can be used to connect to Manticore Search, however they might require certain settings passed in the DSN string as the connector can try running certain SQL commands not implemented yet in Manticore.

JDBC Connector 6.x and above require Manticore Search 2.8.2 or greater and the DSN string should contain the following options:

jdbc:mysql://IP:PORT/DB/?characterEncoding=utf8&maxAllowedPacket=512000&serverTimezone=XXX


In case mysql_version_string is used to suppress the Manticore version, the override version should be smaller than 5.1.1.

.NET MySQL connector uses connection pools by default. To correctly get the statistics of SHOW META, queries along with SHOW META command should be send as one multistatement (SELECT ...;SHOW META). With pooling enabled option Allow Batch=True is required to be added to connection string to allow multistatements.

Server=127.0.0.1;Port=9306;Database=somevalue;Uid=somevalue;Pwd=;Allow Batch=True;

##### ODBC¶

Manticore can be accessed using ODBC. It’s recommended to set charset=UTF8 in the ODBC string. Some ODBC drivers will not like the reported version by the Manticore daemon. This can be overridden with mysql_version_string option.

The SphinxQL protocol can also be used by a MYSQL server to connect to Manticore using FEDERATED engine. For more details about how to connect to Manticore using FEDERATED consult MySQL FEDERATED storage engine support.

#### HTTP API protocol¶

Recommended port for the HTTP protocol is 9308. Please note this port is not assigned by IANA, while there is no known application to use it, it should be checked if it’s available.

There are no special requirements and any HTTP client can be used to connect to Manticore. Some endpoints may require specific headers set, for more information consult HTTP API reference.

### A guide on indexes¶

The Manticore Search daemon can serve multiple data collections, called indexes.

Manticore Search supports two storage index types:

• plain (also called offline or disk) index. Data is indexed once at creation, it supports online rebuilding and online updates for non-text attributes
• RealTime index. Similar to a database table, online updates are possible at any given time

In addition, a special index based on RealTime type, called percolate, can be used to store Percolate Queries.

In the current version, indexes use a schema like a normal database table. The schema can have 3 big types of columns:

• the first column is always an unsigned 64 bit non-zero number, called id. Unlike in a database, there is no mechanism of auto incrementing, so you need to be sure the documents ids are unique
• fulltext fields - they contain indexed content. There can be multiple fulltext fields per index. Fulltext searches can be made on all fields or selective. Currently the original text is not stored, so if it’s required to show their content in search results, a trip to the origin source must be made using the ids (or other identifier) obtained from the search
• attributes - their values are stored and are not used in fulltext matching. Instead they can be used for regular filtering, grouping, sorting. They can be also used in expressions of score ranking.

Field and attribute names must start with a letter and can contain letters, digits and underscore.

The following types can be stored in attributes:

• unsigned 32 bit and signed 64 bit integers
• 32 bit single precision floats
• UNIX timestamps
• booleans
• strings (they can be used just for comparison,grouping or sorting by)
• JSON objects
• multi-value attribute list of unsigned 32-bit integers

Manticore Search supports a storeless index type called distributed which allows searching over multiple indexes. The connected indexes can be local or remote. Distributed indexes allow spreading big data over multiple machines or building high availability setups. As searching over an index is single-threaded, local distributed indexes can be used to make use of multiple CPU cores.

#### Plain indexes¶

Except numeric (that includes MVA) attributes, the rest of the data in a plain index is immutable. If you need to update/add new records you need to perform again a rebuilding. While index is being rebuilt, existing index is still available to serve requests. When new version is ready, a process called rotation is performed which puts the new version online and discards the old one.

The indexing performance process depends on several factors:

• how fast the source can be providing the data
• tokenization settings
• hardware resource (CPU power, storage speed)

In the most simple usage scenario, we would use a single plain index which we rebuild it from time to time.

This implies:

• index is not as fresh as the data from the source
• indexing duration grows with the data

If we want to have the data more fresh, we need to shorten the indexing interval. If indexing takes too much, it can even overlap the time between indexing, which is a major problem. However, Manticore Search can perform a search on multiple indexes. From this, an idea was born to use a secondary index that captures only the most recent updates.

This index will be a lot smaller and we will index it more frequently. From time to time, as this delta index will grow, we will want to “reset” it.

This can be done by either reindexing the main index or merge the delta into the main. The main+delta index schema is detailed at Delta index updates.

As the engine can’t globally do a uniqueness on the document ids, an important thing that needs to be considered is if the delta index could contain updates on existing indexed records in the main index.

For this, there is an option that allows defining a list of document ids which are suppressed by the delta index. For more details, check sql_query_killlist.

#### Real-Time indexes¶

RealTime indexes allow online updates, but updating fulltext data and non-numeric attributes require a full row replace.

The RealTIme index starts empty and you can add, replace, update or delete data in the same fashion as for a database table. The updates are first held into a memory zone, defined by rt_mem_limit. When this gets filled, it is dumped as disk chunk - which as structure is similar with a plain index. As the number of disk chunks increase, the search performance decreases, as the searching is done sequentially on the chunks. To avoid that, there is a command that can merge the disk chunks into a single one - OPTIMIZE INDEX syntax.

Populating a RealTime can be done in two ways: firing INSERTs or converting a plain index to become RealTime. In case of INSERTs, using a single worker (a script or code) that inserts one record at a time can be slow. You can speed this by batching many rows into one and by using multiple workers that perform inserting. Parallel inserts will be faster but also come at using more CPU. The size of the data buffer memory (which we call RAM chunk) also influence the speed of inserting.

#### Local distributed indexes¶

A distributed index in Manticore Search doesn’t hold any data. Instead it acts as a ‘master node’ to fire the demanded query on other indexes and provide merged results from the responses it receives from the ‘node’ indexes. A distributed index can connect to local indexes or indexes located on other servers. In our case, a distributed index would look like:

index_dist {
type = distributed
local = index1
local = index2
...
}


The last step to enable multi-core searches is to define dist_threads in searchd section. Dist_threads tells the engine the maximum number of threads it can use for a distributed index.

#### Remote distributed indexes and high availability¶

index mydist {
type = distributed
agent = box1:9312:shard1
agent = box2:9312:shard2
agent = box3:9312:shard3
agent = box4:9312:shard4
}


Here we have split the data over 4 servers, each serving one of the shards. If one of the servers fails, our distributed index will still work, but we would miss the results from the failed shard.

index mydist {
type = distributed
agent = box1:9312|box5:9312:shard1
agent = box2:9312:|box6:9312:shard2
agent = box3:9312:|box7:9312:shard3
agent = box4:9312:|box8:9312:shard4
}


Now we added mirrors, each shard is found on 2 servers. By default, the master (the searchd instance with the distributed index) will pick randomly one of the mirrors.

The mode used for picking mirrors can be set with ha_strategy. In addition to random, another simple method is to do a round-robin selection ( ha_strategy= roundrobin).

The more interesting strategies are the latency-weighted probabilities based ones. noerrors and nodeads not only that take out mirrors with issues, but also monitor the response times and do a balancing. If a mirror responds slower (for example due to some operations running on it), it will receive less requests. When the mirror recovers and provides better times, it will get more requests.

#### Replication and cluster¶

To use replication define one listen port for SphinxAPI protocol and one listen for replication address and port range in the config. Define data_dir folder for incoming indexes.

searchd {
listen = 9312
listen = 192.168.1.101:9360-9370:replication
data_dir = /var/lib/manticore/
...
}


Create a cluster (via SphinxQL) at the daemon that has local indexes that need to be replicated

CREATE CLUSTER posts


Add these local indexes to cluster

ALTER CLUSTER posts ADD pq_title


All other nodes that want replica of cluster’s indexes should join cluster as

JOIN CLUSTER posts AT '192.168.1.101:9312'


When running queries prepend the index name with the cluster name (posts:).

INSERT INTO posts:pq_title VALUES ( 3, 'test me' )


Now all such queries that modify indexes in the cluster are replicated to all nodes in the cluster.

### A guide on searching¶

There is no difference between RealTime or plain indexes in terms of how you run queries.

The recommended and simplest way to query Manticore is to use the SphinxQL interface. You can access it with any MySQL client or library, just do

$mysql -P9306 -h0  While it implements the MySQL protocol, SphinxQL is not 100% compatible with MySQL syntax. There are specific extensions, like MATCH clause [the most powerful thing in Manticore] or WITHIN GROUP BY and many functions available in MySQL are not implemented (or they are dummy just to allow compatibility with MySQL connector e.g.) or JOINs between indexes which are not supported yet. #### Running queries¶ In the guide of the indexes we already saw an example of a search. In addition to the fulltext match, you can also have attribute filtering, grouping and sorting by attributes or expressions. mysql> SELECT *,weight() FROM myrtindex WHERE MATCH('text') AND gid>20 ORDER BY gid ASC,WEIGHT() DESC; SHOW META; +------+------+----------+ | id | gid | weight() | +------+------+----------+ | 3 | 22 | 2230 | | 2 | 22 | 1304 | | 4 | 33 | 2192 | +------+------+----------+ 3 rows in set (0.00 sec) +---------------+-------+ | Variable_name | Value | +---------------+-------+ | total | 3 | | total_found | 3 | | time | 0.000 | | keyword[0] | text | | docs[0] | 4 | | hits[0] | 7 | +---------------+-------+ 6 rows in set (0.00 sec)  Here we also added a SHOW META command (you can run it in another call, but must be on same session to give information from the query you’ve just executed). For general usage, total_found and time are most useful. Manticore supports LIMIT clause like traditional databases in the format LIMIT [offset,] row_count. If no LIMIT is set, the first 20 rows of the result set are returned. Another non-standard clause is OPTION, which can be used to set various settings for the query. #### Fulltext Matching¶ By default, operator AND is used if multiple keywords are specified. The keywords are searched over all fulltext fields and unless there are other rules, a match is valid when the keywords are found in any of the fulltext fields. So for example ‘search for something’ will give you a match on a document where ‘search’ and ‘for’ are find in ‘title’ field and ‘something’ in ‘content’ field. Restricting the search to certain field(s) can be done with @ operator followed by the name of the field(s), for example @title search for something. Most operators use keyword position relative to document and will give a positive match only if the keywords are found in same field, like proximity, phrase, fied-start/end,NEAR, strict order etc. There are operators for which the keyword position has no influence, like boost operator, exact form modifier or qourum. #### Ranking fulltext matches¶ Manticore offers a powerful way to construct scoring formulas for the fulltext match. There are several building rankers (predefined scoring formulas) with default been proximity_bm25 and custom expressions can be made using the 20+ ranking factors and attributes values if needed. The most important factors are BM25 - an industry retrieval function that ranks the document based on the query terms appearances, it’s a per document factor IDF - inverse document frequency, a numeric statistic that reflect how important a word is to a document in the collection, it is used per field. The IDF values can be used by several ways (as sum, max etc.) LCS - longest common subsequence, in broad terms it gives the proximity (based on keyword positions). Beside the classic ‘lcs’, several derivates are available too. In addition to those, you can use counters on hits or words, boolean factors like exact hit or exact order and document attributes can be used too inside expressions. Several pre-built ranker expressions are available: proximity_bm25, bm25, none, wordcount, proximity, matchany, sph04, expr (custom rankers) and export (same as expr, but stores for output the factor values). They can be changed using the OPTION statement, for example OPTION ranker=bm25. The default proximity_bm25 can be written as custom ranker as OPTION ranker=expr('sum(lcs*user_weight)+bm25'). The user_weight relates to the boost per field, by default all fields are treated equal. For example if you have fields ‘title’ and ‘content’ you might want to give a boost to ‘title’ matching so you would set OPTION field_weights=(title=10, content=1). The ranking score is relative to the query itself as long as it includes metrics that calculate distances between keywords or keywords/document frequencies. In these cases, the values of the score can differ a lot from query to query, so doing any kind of comparison between scores of different queries does not make sense. MySQL [(none)]> SELECT *,weight() FROM myrtindex WHERE MATCH('"more this text"/2') OPTION ranker=proximity_bm25; +------+------+----------+ | id | gid | weight() | +------+------+----------+ | 3 | 22 | 4403 | | 4 | 33 | 3378 | | 2 | 22 | 2453 | | 1 | 11 | 2415 | +------+------+----------+ 4 rows in set (0.00 sec) .. code-block:: none MySQL [(none)]> SELECT *,weight() FROM myrtindex WHERE MATCH('"more this text"/2') OPTION ranker=none; +------+------+----------+ | id | gid | weight() | +------+------+----------+ | 1 | 11 | 1 | | 2 | 22 | 1 | | 3 | 22 | 1 | | 4 | 33 | 1 | +------+------+----------+ 4 rows in set (0.00 sec) .. code-block:: none MySQL [(none)]> SELECT *,weight() FROM myrtindex WHERE MATCH('"more this text"/2') OPTION ranker=expr('sum(1)+gid'); +------+------+----------+ | id | gid | weight() | +------+------+----------+ | 4 | 33 | 35 | | 2 | 22 | 24 | | 3 | 22 | 24 | | 1 | 11 | 13 | +------+------+----------+ 4 rows in set (0.00 sec)  #### Data tokenization¶ Search engines don’t store text as it is. Instead they extract words and create several structures that allows fast full-text searching. From the found words, a dictionary is build, which allows a quick look to discover if the word is present or not in the index. In addition, other structures records the documents and fields in which the word was found (as well as position of it inside a field). All these are used when a full-text match is performed. The process of demarcating and classifying words is called tokenization. The tokenization is applied at both indexing and searching and it operates at character and word level. On the character level, the engine allows only certain characters to pass, this is defined by the charset_table, anything else is replaced with a whitespace (which is considered the default word separator). The charset_table also allows mappings, for example lowercasing or simply replacing one character with another. Beside this, characters can be ignored, blended, defined as a phrase boundary. At the word level, the base setting is the min_word_len which defines the minimum word length in characters to be accepted in the index. A common request is to match singular with plural forms of words. For this, morphology processors can be used. Going further, we might want a word to be matched as another one - because they are synonyms. For this, the wordforms feature can be used, which allows one or more words to be mapped to another one. Very common words can have some unwanted effects on searching, mostly because of their frequency they require lots of computing to process their doc/hit lists. They can be blacklisted with the stopwords features. This helps not only on speeding queries, but also on decreasing index size. A more advanced blacklisting is bigrams, which allows creating a special token between a ‘bigram’ (common) word and an uncommon word. This can speed up several times when common words are used in phrase searches. In case of indexing HTML content, it’s desired to not index also the HTML tags, as they can introduce a lot of ‘noise’ in the index. HTML stripping can be used and can be configured to strip, but index certain tag attributes or completely ignore content of certain HTML elements. Another common text search type is wildcard searching. Wildcard searching is performed at dictionary level. By default, both plain and RT indexes use a dictionary type called keywords. In this mode words are stored as they are, so the size of the index is not affected by enabling wildcarding. When a wildcard search is performed, in the dictionary a lookup is made to find all possible expansions of the wildcarded word. This expansion can be problematic in terms of computation at query time in cases where the expanded word can provide lot of expansions or expansions that have huge hitlists. The penalties are higher in case of infixes, where wildcard is added at the start and end of the words. Even more, usage the expand_keywords setting, which can apply automatically the stars to the input search terms, should be made with care. The plain index also supports a crc dictionary type. With this type, words are not stored as they are, instead a control sum value of words is used. Indexing is much faster in this case compared to keywords mode. Since it would not be possible to do substring search on the CRCs, instead all possible substrings of the words (defined by min_prefix_len or min_infix_len) are also stored. This increase the index size several times when prefix/infix are enabled, but wildcard querying doesn’t suffer performance penalties as it doesn’t need to perform expansions like keywords dictionary. On indexes with crc dictionary it’s not possible to use QSUGGEST feature (since control sums are stored in index instead of actual words) and it’s not possible to convert to RealTime indexes (which only work with keywords dictionary). #### Multi-threaded searching¶ One index may not be enough. When searching, only one search thread (that uses a cpu core) is used for a query. Because of the size of the data or heavy computing queries, we would want to use more than a CPU core per query. To do that, we need to split the index into several smaller indexes. One common way to split the data is to perform a modulo filtering on the document id (like sql_query = SELECT * FROM mytable where id % 4 = 0 [1,2,3]). Having several indexes instead of one means now we can run multiple indexing operations in parallel. Faster indexing comes with a cost: several CPU cores will be used instead of one, there is more pressure on the source (especially if you rebuild all the indexes at once) and multiple threads writing to disk can overload your storage ( you can limit the impact of IO on storage with max_iops and max_iosize directives). Searching over these shards can be done in 2 ways: • one is to simply enumerate them in the query, like SELECT * FROM index0,index1,index2,index3. dist_threads >1 can be used for multi-core processing. • using a local distributed index and dist_threads > 1 (for multi-core processing). #### Grouping and faceting¶ Manticore Search supports grouping by multiple columns or computed expressions. Results can be sorted inside a group with WITHIN GROUP ORDER BY. A particular feature is returning more than one row per group, by using GROUP n BY. Grouping also supports HAVING clause, GROUP_CONCAT and aggregation functions. Manticore Search also supports faceting, which in essence is a set of group by applied on the same result set. mysql> SELECT * FROM myindex WHERE MATCH('some thing') and afilter=1 FACET attr_1 FACET_2 attr_2; +------+---------+----------+----------+ | id | attr_1 | attr_2 | afilter | +------+------+-------------+----------+ | 4 | 33 | 35 | 1 | ........ +------+------------+ | attr_1 count(*) | +------+------------+ | 4 | 33 | ........ +------+------------+ | attr_2 count(*) | +------+------------+ | 10 | 1 |  In return you get a multiple result set, where the first is the result set of the query and the rest are the facet results. #### Functions¶ GEODIST function can be used to calculate distance between 2 geo coordinates. The result can be used for sorting. mysql> SELECT *, GEODIST(0.65929812494086, -2.1366023996942, latitude, longitude, {in=rad, out=mi,method=adaptive}) AS distance FROM geodemo WHERE distance < 10000 ORDER BY distance ASC LIMIT 0,100;  In addition, polygon calculation can be made, including geo polygon that takes into account Earth’s curvature. mysql> SELECT *, CONTAINS(GEOPOLY2D(40.95164274496,-76.88583678218,41.188446201688,-73.203723511772,39.900666261352,-74.171833538046,40.059260979044,-76.301076056469),latitude_deg,longitude_deg) AS inside FROM geodemo WHERE inside=1;  Manticore Search also supports math, date and aggregation functions which are documented at Expressions, functions, and operators. Special functions ALL() and ANY() can be used to test elements in an array from a JSON attribute or MVA. #### Highlighting¶ Highlighting allows to get a list of fragments from documents (called snippets) which contain the matches. The snippets are used to improve the readability of search results to end users. Snippeting can be made with the CALL SNIPPETS statement. The function needs the texts that will be highlighted, the index used (for it’s tokenization settings), the query used and optionally a number of settings can be applied to tweak the operation. mysql> CALL SNIPPETS('this is my hello world document text I am snippeting now', 'myindex', 'hello world', 1 as query_mode, 5 as limit_words); +------------------------------------------------+ | snippet | +------------------------------------------------+ | ... my <b>hello world</b> document text ... | +------------------------------------------------+ 1 row in set (0.00 sec)  #### Tokenizer tester¶ CALL KEYWORDS provides a way to check how keywords are tokenized or to retrieve the tokenized forms of particular keywords.. Beside debug/testing, CALL KEYWORDS can be used for transliteration. For example we can have a template index which maps characters from cyrillic to latin. We can use CALL KEYWORDS to get the latin form of a word written in cyrillic. mysql> call keywords ('ran','myindex'); +------+-----------+------------+ | qpos | tokenized | normalized | +------+-----------+------------+ | 1 | ran | run | +------+-----------+------------+ 1 row in set (0.00 sec)  #### Suggested words¶ CALL SUGGEST enabled getting suggestions or corrections of a given words. This is useful to implement ‘did you mean …’ functionality. CALL SUGGEST requires an index with full wildcarding (infixing) enabled. Suggestion is based on the index dictionary and uses Levenshtein distance. Several options are available to allow tweaking and the output provide, beside distance, a document count for each word. In case at input there is more than one word, CALL SUGGEST will only process the first word, while CALL QSUGGEST will only process the last word and ignore the rest. mysql> call suggest('sarch','myindex'); +---------+----------+------+ | suggest | distance | docs | +---------+----------+------+ | search | 1 | 6071 | | arch | 1 | 20 | | march | 1 | 10 | | sarah | 1 | 4 | +---------+----------+------+ 4 rows in set (0.00 sec)  #### Percolate queries¶ The regular workflow is store index document and match against them a query. However, sometimes it’s desired to check if new content matches an existing set of queries. Running the queries over the index each time a document is added can be inefficient. Instead, it would be faster if queries were stored in a index and the new documents are tested against the stored queries. Also called inverse search, this is used for for signaling in monitoring systems or news aggregation. For this, a special index is used called percolate, which is similar with a RealTime index. The queries are stored in a percolate index and CALL PQ can test one or more documents if they match against the stored queries. mysql> INSERT INTO index_name VALUES ( 'this is a query'); mysql> INSERT INTO index_name VALUES ( 'this way'); mysql> CALL PQ ('index_name', ('multiple documents', 'go this way'), 0 as docs_json );  #### Search performance¶ To debug and understand why a search is slow, information is provided by commands SHOW PROFILE, SHOW PLAN and SHOW META. Tokenization and search expression can have a big impact on the search speed. They can generate requesting a lot of data from index components and/or require heavy computation (like merging big lists of hits). An example is using wildcarding on very short words, like 1-2 characters. An index is not fully loaded by default into memory. Only several components are, such as dictionary or attributes (which can be set to not be loaded). The rest will be loaded when queries are made. Operating systems will cache read files from the storage. If there is plenty of RAM, an index can be cached enterily as searches are made. If the index is not cached, a slow storage will impact searches. Also, the load time of an index is influenced by how fast components can be loaded into RAM. For small indexes this is not a problem, but in case of huge indexes it can take minutes until an index is ready for searches. Queries can also be CPU-bound. This is because index is too big or it’s settings or search perform heavy computation. If an index grows big, it should be split to allow multi-core searching as explained in previous guide. If we talk about big data, one server may not be enough and we need to spread our indexes over more than one server. Servers should be as close as possible (at least same data center), as the network latencies between master and nodes will affect the query performance. ## Installation¶ ### Installing Manticore packages on Debian and Ubuntu¶ Supported releases: • Debian • 8.0 (jessie) • 9.0 (stretch) • Ubuntu • 14.04 LTS (trusty) • 16.04 LTS (xenial) • 18.04 LTS (bionic) Supported platforms: • x86 • x86_64 You can install Manticore with command: $ wget https://github.com/manticoresoftware/manticore/releases/download/2.4.1/manticore_2.4.1-171017-3b31a97-release-stemmer.jessie_amd64-bin.deb
$sudo dpkg -i manticore_2.4.1-171017-3b31a97-release-stemmer.jessie_amd64-bin.deb  Manticore package depends on zlib and ssl libraries, nothing else is strictly required. However if you plan to use ‘indexer’ tool to create indexes from different sources, you’ll need to install appropriate client libraries. To know what exactly libraries, run indexer tool from Manticore and look at the top of it’s output: $ indexer
Manticore 2.4.1 4258276@171019 id64-beta
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017, Manticore Software LTD (http://manticoresearch.com)

Built by gcc/clang v 6.3.0,

Built on Linux d2a57137d4f5 4.8.0-45-generic #48~16.04.1-Ubuntu SMP Fri Mar 24 12:46:56 UTC 2017 x86_64 GNU/Linux
Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DDL_MYSQL=1 -DMYSQL_LIB=libmariadbclient.so.18 -DMYSQL_CONFIG_EXECUTABLE=/usr/bin/mysql_config -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DSPLIT_SYMBOLS=ON -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=ON -DWITH_ICONV=ON -DWITH_MYSQL=ON -DWITH_ODBC=ON -DWITH_PGSQL=ON -DWITH_RE2=ON -DWITH_STEMMER=ON -DWITH_ZLIB=ON


Here you can see mentions of libodbc.so.2, libexpat.so.1, libmariadbclient.so.18, and libpq.so.5.

Below is the reference table with list of all client libraries for different debian/ubuntu distributions:

Distr Mysql PostgresQL Xmlpipe Unixodbc
trusty libmysqlclient.so.18 libpq.so.5 libexpat.so.1 libodbc.so.1
xenial libmysqlclient.so.20 libpq.so.5 libexpat.so.1 libodbc.so.2
bionic libmysqlclient.so.20 libpq.so.5 libexpat.so.1 libodbc.so.2
wheezy libmysqlclient.so.18 libpq.so.5 libexpat.so.1 libodbc.so.1
jessie libmysqlclient.so.18 libpq.so.5 libexpat.so.1 libodbc.so.2

To find the packages which provide the libraries you can use, for example apt-file:

$apt-file find libmysqlclient.so.20 libmysqlclient20: /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 libmysqlclient20: /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.2.0 libmysqlclient20: /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.6  Note, that you need only libs for types of sources you’re going to use. So if you plan to make indexes only from mysql source, then install only lib for mysql client (in case above - libmysqlclient20). Finally install necessary packages: $ sudo apt-get install libmysqlclient20 libodbc1 libpq5 libexpat1


If you aren’t going to use indexer tool at all, you don’t need find and install any libraries.

After preparing configuration file (see Quick tour), you can start searchd daemon:

$systemctl start manticore  To enable Manticore at boot: $ systemctl enable manticore


### Installing Manticore packages on RedHat and CentOS¶

Supported releases:

• CentOS 6 and RHEL 6
• CentOS 7 and RHEL 7

Supported platforms:

• x86
• x86_64

Manticore package depends on zlib and ssl libraries, nothing else is strictly required. However if you plan to use ‘indexer’ tool to create indexes from different sources, you’ll need to install appropriate client libraries. Use yum to download and install these dependencies:

$yum install mysql-libs postgresql-libs expat unixODBC  Note, that you need only libs for types of sources you’re going to use. So if you plan to make indexes only from mysql source, then installing ‘mysql-libs’ will be enough. If you don’t going to use ‘indexer’ tool at all, you don’t need to install these packages. Download RedHat RPM from Manticore website and install it: $ wget https://github.com/manticoresoftware/manticore/releases/download/2.4.1/manticore-2.4.1-171017-3b31a97-release-stemmer-rhel7-bin.rpm
$rpm -Uhv manticore-2.4.1-171017-3b31a97-release-stemmer-rhel7-bin.rpm  After preparing configuration file (see Quick tour), you can start searchd daemon: $ systemctl start searchd


To enable Manticore at boot:

$systemctl enable searchd  ### Installing Manticore on Windows¶ To install on Windows, you need to download the zip package and unpack it first in a folder. In the following example we’ll consider folder C:\Manticore where we unpack the zip content. cd C:\Manticore unzip manticore-2.4.1-171017-3b31a97-release-pgsql-stemmer-x64-bin.zip  The zip comes with 2 sample configurations: sphinx.conf.in and sphinx-min.conf.in. The latter is a stripped-down of comments version of the first. The configuration contains a @CONFIGDIR@ string which needs to be replaced. The @CONFIGDIR@ is the root directory of data and log folders (first is used as location for indexes, second for logs). The zip package comes with these folders, so they will be available at the location where you unzipped the package. If you want to use a different location, the two folders must be created there. Install the searchd system as a Windows service: C:\Manticore\bin> C:\Manticore\bin\searchd --install --config C:\Manticore\sphinx.conf.in --servicename Manticore  Make sure to use the full path of the configuration file, otherwise searchd.exe will not be able to know the location of it when it’s started as service. After installation, the service can be started from the Services snap-in of the Microsoft Management Console. Once started you can access Manticore using the mysql cli: C:\path\to\mysql> mysql -P9306 -h127.0.0.1  (note that in most example, we use -h0, on Windows you need to use localhost or 127.0.0.1 for the local host.) ### Running Manticore Search in a Docker Container¶ Docker images of Manticore Search are hosted publicly on Docker Hub at https://hub.docker.com/r/manticoresearch/manticore/. For more information about using Docker, see the Docker Docs. The searchd daemon runs in nodetach mode inside the container. Default configuration includes a simple Real-Time index and listens on the default ports (9306 for SphinxQL, 9312 for SphinxAPI and 9308 for HTTP). The image comes with MySQL and PostgreSQL client libraries for indexing data from these databases, as well as with the expat library for XML indexing. #### Starting a Manticore Search instance in a container¶ To start a container running the latest release of Manticore Search run: docker run --name manticore -p 9306:9306 -d manticoresearch/manticore  Operations with utility tools over running daemon can be made with docker exec command: docker exec -it manticore indexer --all --rotate  To stop the Manticore Search container you can simply do: docker stop manticore  or (managed stop with no hard-killing): docker exec -it manticore searchd --stopwait  Please note that any indexed data or configuration change made is lost if the container is stopped. For persistence, you need to mount the configuration and data folders. #### Mounting points¶ The configuration folder inside the image is the usual /etc/sphinxseach. Index files are located at /var/lib/manticore/data and logs at /var/lib/manticore/log. For persistence, mount these points to your local folders. docker run --name manticore -v /path/to/config/:/etc/sphinxsearch/ -v /path/to/data/:/var/lib/manticore/data -v /path/to/logs/:/var/lib/manticore/log -p 9306:9306 -d manticoresearch/manticore  ### Compiling Manticore from source¶ #### Required tools¶ • a working compiler • on Linux - GNU gcc (4.7.2 and above) or clang can be used • on Windows - Microsoft Visual Studio 2015 and above (community edition is enough) • on Mac OS - XCode • cmake - used on all plaftorms (version 3.13 or above) #### Required libraries/packages on Linux¶ • Development version of ssl lib. Usually comes in package named like libssl-dev or openssl-devel. • Development version of boost. On debian packages libboost-system-dev and libboost-program-options-dev are enough; on redhat it is boost-devel. #### Optional dependencies¶ • git, flex, bison - needed if the sources are from cloned repository and not the source tarball • development version of MySQL client for MySQL source driver • development version of unixODBC for the unixODBC source driver • development version of libPQ for the PostgreSQL source driver • development version of libexpat for the XMLpipe source driver • RE2 (bundled in the source tarball) for regexp_filter feature • lib stemmer (bundled in the source tarball ) for additional language stemmers #### General building options¶ For compiling latest version of Manticore, recommended is checkout the latest code from the github repositiory. Alternative, for compiling a certain version, you can either checked that version from github or use it’s respective source tarball. In last case avoid to use automatic tarballs from github (named there as ‘Source code’), but use provided files as manticore-2.4.1-171017-3b31a97-release.tar.gz. When building from git clone you need packages git, flex, bison. When building from tarball they are not necessary. This requirement may be essential to build on Windows. $ git clone https://github.com/manticoresoftware/manticore.git

$wget https://github.com/manticoresoftware/manticore/releases/download/2.4.1/manticore-2.4.1-171017-3b31a97-release.tar.gz$ tar zcvf manticore-2.4.1-171017-3b31a97-release.tar.gz


Next step is to configure the building with cmake. Available list of configuration options:

• CMAKE_BUILD_TYPE - can be Debug , Release , MinSizeRel and RelWithDebInfo (default).

• SPLIT_SYMBOLS (bool) - specify whenever to create separate files with debugging symbols. In the default build type,RelWithDebInfo, the binaries include the debug symbols. With this option specified, the binaries will be stripped of the debug symbols , which will be put in separate files

• USE_BISON, USE_FLEX (bool) - enabled by default, specifies whenever to enable bison and flex tools

• LIBS_BUNDLE - filepath to a folder with different libraries. This is mostly relevant for Windows building

• WITH_STEMMER (bool) - specifies if the build should include the libstemmer library. The library is searched in several places, starting with

• libstemmer_c folder in the source directory
• common system path. Please note that in this case, the linking is dynamic and libstemmer should be available system-wide on the installed systems
• libstemmer_c.tgz in LIBS_BUNDLE folder.
• download from snowball project website. This is done by cmake and no additional tool is required
• NOTE: if you have libstemmer in the system, but still want to use static version, say, to build a binary for a system without such lib, provide WITH_STEMMER_FORCE_STATIC=1 in advance.
• WITH_RE2 (bool) - specifies if the build should include the RE2 library. The library can be taken from the following locations:

• in the folder specified by WITH_RE2_ROOT parameters
• in libre2 folder of the Manticore sources
• system wide search, while first looking for headers specified by WITH_RE2_INCLUDES folder and the lib files in WITH_RE2_LIBS folder
• check presence of master.zip in the LIBS_BUNDLE folder
• NOTE: if you have RE2 in the system, but still want to use static version, say, to build a binary for a system without such lib, provide WITH_RE2_FORCE_STATIC=1 in advance.
• WITH_EXPAT (bool) enabled compiling with libexpat, used XMLpipe source driver

• WITH_MYSQL (bool) enabled compiling with MySQL client library, used by MySQL source driver. Additional parameters WITH_MYSQL_ROOT, WITH_MYSQL_LIBS and WITH_MYSQL_INCLUDES can be used for custom MySQL files

• WITH_ODBC (bool) enabled compiling with ODBC client library, used by ODBC source driver

• WITH_PGSQL (bool) enabled compiling with PostgreSQL client library, used by PostgreSQL source driver

• DISTR_BUILD - in case the target is packaging, it specifies the target operating system. Supported values are: centos6, centos7, wheezy, jessie, stretch, trusty, xenial, bionic, macos, default.

#### Compiling on UNIX systems¶

To install all dependencies on Debian/Ubuntu:

$apt-get install build-essential cmake unixodbc-dev libpq-dev libexpat-dev libmysqlclient-dev libssl-dev libboost-system-dev libboost-program-options-dev git flex bison  Note: on Debian 9 (stretch) package libmysqlclient-dev is absent. Use default-libmysqlclient-dev there instead. To install all dependencies on CentOS/RHEL: $ yum install gcc gcc-c++ make cmake mysql-devel expat-devel postgresql-devel unixODBC-devel openssl-devel boost-devel rpm-build systemd-units  git flex bison


(git, flex, bison doesn’t necessary if you build from tarball)

RHEL/CentOS 6 ship with a old version of the gcc compiler, which doesn’t support -std=c++11 flag, for compiling use devtools repository:

$wget http://people.centos.org/tru/devtools-2/devtools-2.repo -O /etc/yum.repos.d/devtools-2.repo$ yum upgrade -y
$yum install -y devtoolset-2-gcc devtoolset-2-binutils devtoolset-2-gcc-c++$ export PATH=/opt/rh/devtoolset-2/root/usr/bin:$PATH  Manticore uses cmake for building. We recommend to use a folder outside the sources to keep them clean. $ mkdir build
$cd build$ cmake3 -D WITH_MYSQL=1 -DWITH_RE2=1 ../manticore


or if we use sources from tarball:

$cmake3 -D WITH_MYSQL=1 -DWITH_RE2=1 ../manticore-2.4.1-171017-3b31a97-release  To simply compile: $ make -j4


This will create the binary files, however we want to either install Manticore or more convenient to create a package. To install just do

$make -j4 install  For packaging use package $ make -j4 package


By default, if no operating system was targeted, package will create only a zip with the binaries. If, for example, we want to create a deb package for Debian Jessie, we need to specify to cmake the DISTR_BUILD parameter:

$cmake3 -DDISTR_BUILD=jessie ../manticore$ make -j4 package


This will create 2 deb packages, a manticore-x.x.x-bin.deb and a manticore-x.x.x-dbg.deb which contains the version with debug symbols. Another possible target is tarball , which create a tar.gz file from the sources.

#### Compiling on Windows¶

For building on Windows you need:

• Visual Studio
• Cmake for Windows
• Expat, MySQL and PostgreSQL in bundle directory.

If you build from git clone, you also need to provide git, flex, bison tools. They may be fond in cygwin framework. When building from tarball these tools are not necessary.

For a simple building on x64:

C:\build>"%PROGRAMW6432%\CMake\bin\cmake.exe" -G "Visual Studio 14 Win64" -DLIBS_BUNDLE="C:\bundle" "C:\manticore"
C:\build>"%PROGRAMW6432%\CMake\bin\cmake.exe" -DWITH_PGSQL=1 -DWITH_RE2=1 -DWITH_STEMMER=1 .
C:\build>"%PROGRAMW6432%\CMake\bin\cmake.exe" --build . --target package --config RelWithDebInfo


#### Recompilation (update)¶

If you didn’t change path for sources and build, just move to you build folder and run:

cmake3 .
make clean
make


If by any reason it doesn’t work, you can delete file CMakeCache.txt located in build folder. After this step you have to run cmake again, pointing to source folder and configuring the options.

If it also doesn’t help, just wipe out your build folder and begin clean compiling from sources

### Quick Manticore usage tour¶

We are going to use SphinxQL protocol as it’s the current recommended way and it’s also easy to play with. First we connect to Manticore with the normal MySQL client:

$mysql -h0 -P9306  The default configuration comes with a sample Real-Time. A first step to see it in action is to add several documents to it, then you can start perform searches: mysql> INSERT INTO rt VALUES (1, 'this is', 'a sample text', 11); Query OK, 1 row affected (0.00 sec) mysql> INSERT INTO rt VALUES (2, 'some more', 'text here', 22); Query OK, 1 row affected (0.00 sec) mysql> INSERT INTO rt VALUES (3, 'more about this text', 'can be found in this text', 22); Query OK, 1 row affected (0.00 sec)  mysql> SELECT *,weight() FROM rt WHERE MATCH('text') ORDER BY WEIGHT() DESC; +------+------+----------+ | id | gid | weight() | +------+------+----------+ | 3 | 22 | 2252 | | 1 | 11 | 1319 | | 2 | 22 | 1319 | +------+------+----------+ 3 rows in set (0.00 sec)  In the sample configuration there is also a plain index with MySQL source, which needs to be indexed first in order to start using it. First, we populate the sample table in MySQL: mysql> create database test;$ mysql -u test <  /usr/share/doc/manticore/example-conf/example.sql


The sample config uses a test with no password for connecting to MySQL. Adjust the credentials, then index:

$sudo -u manticore indexer -c /etc/sphinxsearch/sphinx.conf test1 --rotate Manticore 2.3.3 9b7033e@170806 master...origin/master-id64-dev Copyright (c) 2001-2016, Andrew Aksyonoff Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com) Copyright (c) 2017, Manticore Software LTD (http://manticoresearch.com) using config file '/etc/sphinxsearch/sphinx.conf'... indexing index 'test1'... collected 4 docs, 0.0 MB sorted 0.0 Mhits, 100.0% done total 4 docs, 193 bytes total 0.002 sec, 81503 bytes/sec, 1689.18 docs/sec total 4 reads, 0.000 sec, 8.1 kb/call avg, 0.0 msec/call avg total 12 writes, 0.000 sec, 0.1 kb/call avg, 0.0 msec/call avg rotating indices: successfully sent SIGHUP to searchd (pid=2947).  Now let’s run several queries: mysql> SELECT *, WEIGHT() FROM test1 WHERE MATCH('"document one"/1');SHOW META; +------+----------+------------+----------+ | id | group_id | date_added | weight() | +------+----------+------------+----------+ | 1 | 1 | 1502280778 | 2663 | | 2 | 1 | 1502280778 | 1528 | +------+----------+------------+----------+ 2 rows in set (0.00 sec) +---------------+----------+ | Variable_name | Value | +---------------+----------+ | total | 2 | | total_found | 2 | | time | 0.000 | | keyword[0] | document | | docs[0] | 2 | | hits[0] | 2 | | keyword[1] | one | | docs[1] | 1 | | hits[1] | 2 | +---------------+----------+ 9 rows in set (0.00 sec)  mysql> SET profiling=1;SELECT * FROM test1 WHERE id IN (1,2,4);SHOW PROFILE; Query OK, 0 rows affected (0.00 sec) +------+----------+------------+ | id | group_id | date_added | +------+----------+------------+ | 1 | 1 | 1502280778 | | 2 | 1 | 1502280778 | | 4 | 2 | 1502280778 | +------+----------+------------+ 3 rows in set (0.00 sec) +--------------+----------+----------+---------+ | Status | Duration | Switches | Percent | +--------------+----------+----------+---------+ | unknown | 0.000059 | 4 | 44.70 | | net_read | 0.000001 | 1 | 0.76 | | local_search | 0.000042 | 1 | 31.82 | | sql_parse | 0.000012 | 1 | 9.09 | | fullscan | 0.000001 | 1 | 0.76 | | finalize | 0.000007 | 1 | 5.30 | | aggregate | 0.000006 | 2 | 4.55 | | net_write | 0.000004 | 1 | 3.03 | | eval_post | 0.000000 | 1 | 0.00 | | total | 0.000132 | 13 | 0 | +--------------+----------+----------+---------+ 10 rows in set (0.00 sec)  mysql> SELECT id, id%3 idd FROM test1 WHERE MATCH('this is | nothing') GROUP BY idd;SHOW PROFILE; +------+------+ | id | idd | +------+------+ | 1 | 1 | | 2 | 2 | | 3 | 0 | +------+------+ 3 rows in set (0.00 sec) +--------+----------+----------+---------+ | Status | Duration | Switches | Percent | +--------+----------+----------+---------+ | total | 0.000000 | 0 | 0 | +--------+----------+----------+---------+ 1 row in set (0.00 sec)  mysql> SELECT id FROM test1 WHERE MATCH('is this a good plan?');SHOW PLAN\G Empty set (0.00 sec) *************************** 1. row *************************** Variable: transformed_tree Value: AND( AND(KEYWORD(is, querypos=1)), AND(KEYWORD(this, querypos=2)), AND(KEYWORD(a, querypos=3)), AND(KEYWORD(good, querypos=4)), AND(KEYWORD(plan, querypos=5))) 1 row in set (0.00 sec)  mysql> SELECT COUNT(*) c, id%3 idd FROM test1 GROUP BY idd HAVING COUNT(*)>1; +------+------+ | c | idd | +------+------+ | 2 | 1 | +------+------+ 1 row in set (0.00 sec)  mysql> SELECT COUNT(*) FROM test1; +----------+ | count(*) | +----------+ | 4 | +----------+ 1 row in set (0.00 sec)  mysql> CALL KEYWORDS ('one two three', 'test1', 1); +------+-----------+------------+------+------+ | qpos | tokenized | normalized | docs | hits | +------+-----------+------------+------+------+ | 1 | one | one | 1 | 2 | | 2 | two | two | 1 | 2 | | 3 | three | three | 0 | 0 | +------+-----------+------------+------+------+ 3 rows in set (0.00 sec)  ## Indexing¶ ### Indexes¶ To be able to answer full-text search queries fast, Manticore needs to build a special data structure optimized for such queries from your text data. This structure is called index; and the process of building index from text is called indexing. An index identifier must be a single word, that can contain letters, numbers and underscores. It must start with a letter. Different index types are well suited for different tasks. For example, a disk-based tree-based index would be easy to update (ie. insert new documents to existing index), but rather slow to search. Manticore architecture allows internally for different index types, or backends, to be implemented comparatively easily. Manticore provides 2 different backends: a disk index backend, and a RT (realtime) index backend. #### Plain indexes¶ Disk indexes are designed to provide maximum indexing and searching speed, while keeping the RAM footprint as low as possible. That comes at a cost of text index updates. You can not update an existing document or incrementally add a new document to a disk index. You only can batch rebuild the entire disk index from scratch. (Note that you still can update document’s attributes on the fly, even with the disk indexes.) This “rebuild only” limitation might look as a big constraint at a first glance. But in reality, it can very frequently be worked around rather easily by setting up multiple disk indexes, searching through them all, and only rebuilding the one with a fraction of the most recently changed data. See Live index updates for details. #### Real-Time indexes¶ RT indexes enable you to implement dynamic updates and incremental additions to the full text index. RT stands for Real Time and they are indeed “soft realtime” in terms of writes, meaning that most index changes become available for searching as quick as 1 millisecond or less, but could occasionally stall for seconds. (Searches will still work even during that occasional writing stall.) Refer to Real-time indexes for details. #### Distributed indexes¶ Manticore supports so-called distributed indexes. Compared to disk and RT indexes, those are not a real physical backend, but rather just lists of either local or remote indexes that can be searched transparently to the application, with Manticore doing all the chores of sending search requests to remote machines in the cluster, aggregating the result sets, retrying the failed requests, and even doing some load balancing. See Distributed searching for a discussion of distributed indexes. #### Templates indexes¶ Template indexes are indexes with no storage backend. They can be used operations that involve only data from input, like keywords and snippets generation. #### Percolate indexes¶ Percolate indexes are special Real-Time indexes that store queries instead of documents. They are used for prospective searches ( or “search in reverse”). Refer to Percolate query for more details. There can be as many indexes per configuration file as necessary. indexer utility can reindex either all of them (if --all option is specified), or a certain explicitly specified subset. searchd utility will serve all the specified indexes, and the clients can specify what indexes to search in run time. #### Index files¶ Each index consists of a number of files. Small index components are fully loaded into memory. Big index components are read from disk as needed. Currently these use seek+read or mmap() to retrieve data from disk. Attribute components are opened and mapped with mmap(). They can be loaded fully in memory or left on disk and read when needed. Plain indexes and RealTime indexes chunks: Extension Stores Memory management spa scalar attrs mmap(), also see access_plain_attrs spd document lists read from disk and may be cached by OS or mmaped, also see access_doclists spi dictionary always loaded in memory sph index/chunk header always loaded in memory spk kill list loaded and discarded [1] spl index lock file on disk only spm row map mmap() sphi secondary index histograms always loaded in memory spt docid lookups mmap() spp keyword positions read from disk and may be cached by OS or mmaped, also see access_hitlists spb var-length attrs mmap(), also see access_blob_attrs [1] Kill lists - loaded in memory at startup and discarded after they are applied to targets RealTime indexes also have: Extension Stores Memory management kill RT kill [1] on disk only meta RT header always loaded in memory lock RT lock file on disk only ram RAM chunk copy [2] on disk only [1] RT kill - documents that are REPLACEd get cleared when the RAM chunk is dumped as a disk chunk. [2] RAM chunk copy - created when the RAM chunk is flushed to disk. Cleared when the RAM chunk is dumped as a disk chunk. #### Accessing index files¶ The daemon uses two access modes to read index data - seek+read and mmap. In seek+read mode the daemon performs system call pread(2) to read document lists and keyword positions, i.e. spd and spp files. Internal read buffers are used to optimize reading. The size of these buffers can be tuned with options read_buffer_docs and read_buffer_hits. There is also preopen option that allows to control the amount of files opened by daemon at start. In mmap access mode the search daemon just maps index’s file into memory with mmap(2) system call and OS caches file content by itself. Options read_buffer_docs and read_buffer_hits have no effect for corresponding files in this mode. This reader could be used for scalar (int, float, boolean, timestamp) attributes, var-length (string, mva, json) attributes, document lists and keyword positions, i.e. spa, spb, spd and spp files. The mmap reader can also lock index’s data in memory via mlock(2) privileged call which prevents swapping out of the cached data to disk by OS. To control what access mode will be used access_plain_attrs, access_blob_attrs, access_doclists and access_hitlists options are available with the following values: • file daemon reads index file from disk with seek+read using internal buffers on file access • mmap daemon maps index file into memory and OS caches up its contents on file access • mmap_preread daemon maps index file into memory and a background thread reads it once to warm up the cache • mlock daemon maps index file into memory and then issues mlock system call to cache up the file contents and lock it into memory to prevent it being swapped out Here is a table which can help you select your desired mode:  index part keep it on disk keep it in memory cached in memory on daemon start lock it in memory .spa (plain attributes) access_plain_attrs=mmap - the file will mapped to RAM, but your OS will decide whether to really load it to RAM or not and can easily swap it out (default) access_plain_attrs = mmap_preread (default) access_plain_attrs = mlock .spb (blob attributes) access_blob_attrs=mmap - the file will mapped to RAM, but your OS will decide whether to really load it to RAM or not and can easily swap it out (default) access_blob_attrs = mmap_preread (default) access_blob_attrs = mlock .spd (doc lists) access_doclists = file (default) access_doclists = mmap, may be still swapped out by OS no no .spp (hit lists) access_hitlists = file (default) access_hitlists = mmap, may be still swapped out by OS no no There’s also a searchd command line option --force-preread that instructs the daemon to wait until the attribute files are read prior to starting accepting incoming connections. Starting daemon with this option allows to make sure that all mapped index files will be ‘warmed’ by touching every page of the maps. OS, in turn, usually tries to keep all ‘warm’ data in the disk cache, providing maximum performance. The recommendations are: • If search performance is very important and you have enough memory - use mlock for attributes and mmap for doclists/hitlists. Be aware mlock is a privileged system call and the user running searchd should have enough privileges. Read here for details • If you can’t afford lower performance on start and ready to wait longer on start until it’s warmed up - use –force-preread • If you want searchd to be able to restart faster - stay with mmap_preread • If you want to save RAM - do not use mlock, then your OS will decide what should be in memory at any given moment of time depending on what is read from disk more frequently • If search performance doesn’t matter at all and you want to save maximum RAM - use access_doclists/access_hitlists=file and access_plain_attrs/access_blob_attrs=mmap The default mode is to mmap and pre-read attributes and access doclists/hitlists directly from disk which provides decent search performance, optimal memory usage and faster searchd restart in most cases. #### Operations on indexes¶ ##### Declaration¶ Plain indexes can only be created by indexer tool. If a plain index is only declared in configuration,but not created, the daemon will print a warning about that. It must be also noted that the daemon requires at least one index of type RT, percolate or plain in order to start. Real-Time, percolate and template indexes can be declared in the configuration and they will be created (with empty data) at daemon start. ##### Loading or discarding indexes¶ At startup, daemon will try to load and make available all indexes found in the configuration file. HUP signal can be used to make the daemon reload the configuration. This way new indexes can be loaded or existing indexes can be discarded while the daemon is running. Changing the type of an index, for example from template to Real-Time, can also be performed during a configuration reload. Alternative to signaling HUP to searchd daemon, the RELOAD INDEXES SphinxQL command can be used. Refreshing a plain index already loaded by daemon requires running indexer with –rotate parameter. In this case, a new version of the plain index is created and when ready, a HUP is send to daemon, which will load the new version of the index in the memory and discard the old one. ##### Index changes¶ Index schema can be changed on-the-fly in case of attribute. Full-text fields however require re-creating the index. Change of tokenization settings requires a remaking in case of plain indexes. For Real-Time indexes, these can be made on-the-fly using ALTER RECONFIGURE but they will affect only new content added to index, as it’s not possible yet to re-tokenize already indexed texts. Some settings like access_plain_attrs, which don’t alter in any way the index, don’t require an index rebuild, just a reload. ### Data Types¶ The following data types are supported in Manticore Search: #### Text¶ It is the full-text field part of the index. The content of these fields is indexed and not stored in the original form. The text is passed through an analyzer pipeline that converts the text to words, applies morphology transformations etc. Full-text fields can only be used in MATCH() clause, they are not returned in the result set and cannot be used for sorting or aggregation. Words are stored in an inverted index along with references to the fields they belong and positions in the field. This allows to search a word inside each field and to use advanced operators like proximity. #### String¶ Unlike full-text fields, the string attributes are stored as they are received and cannot be used in full-text searches. Instead they are returned in results, they can be used in WHERE clause for comparison filtering or REGEX and they can be used for sorting and aggregation. In general it’s recommended to not store large texts in string attributes, but use string attributes for metadata like names, titles, tags, keys. #### Integer¶ Integer type allows 32 bit unsigned integer values. Integers can be stored in shorter sizes than 32 bit by specifying a bit count. For example if we want to store a numeric value which we know is not going to be bigger than 8, the integer can be defined as sql_attr_uint = myattr:3  Bitcount integers perform slowly than the full size ones, but they require less RAM. The bitcount integers are saved in 32-bit chunks, to save space they should be grouped at the end of attributes definitions (otherwise a bitcount integer between 2 full-size integers will occupy 32 bits as well). #### Big Integer¶ Big integers are 64-bit wide signed integers. #### Timestamps¶ Timestamp allows to represent a unix timestamp which is stored as a 32-bit integer. For timestamps a family of date and time functions are available. #### Float¶ Real numbers can be stored as 32-bit IEEE 754 single precision floats. Unlike integer types, equal comparison of floats is forbidden due to rounding errors. A near equal can be used instead, by checking the absolute error margin: SELECT ABS(a-b) <= 0.00001  where 0.00001 is a chosen accepted tolerance. Another alternative, which can also be used to perform IN(attr,val1,val2,val3) is to compare floats as integers by choosing a multiplier factor and convert the floats to integers in operations. For example: SELECT IN(attr,2.0,2.5,3.5) ...  can be made as SELECT IN(CEIL(attr*100),200,250,350) ....  #### JSON¶ Allows storing JSON objects for schema-less data. JSON properties can be used in most operations and special functions like ALL(), ANY(),:ref:GREATEST() <expr-func-greatest>,:ref:LEAST() <expr-func-least> and INDEXOF() allow traversal of property arrays. Text properties are treated same as strings (so it’s not possible to use them in full-text matches). #### Multi-value integer¶ It’s a special type that allows storing variable-length lists of 32-bit unsigned integers. It can be used to store one-to-many numeric values like tags, product categories, properties. It supports filtering and aggregation, but not sorting. Filtering can made of condition that requires at least one element to pass (using ANY()) or all (using ALL()). Information like least or greatest element and length of the list can be extracted. #### Multi-value big integer¶ It’s a special type that allows storing variable-length lists of 64-bit signed integers. It has the same functionality as multi-value integer. ### Full-text fields¶ Full-text fields (or just fields for brevity) are the textual document contents that get indexed by Manticore, and can be (quickly) searched for keywords. Fields are named, and you can limit your searches to a single field (eg. search through “title” only) or a subset of fields (eg. to “title” and “abstract” only). Manticore index format generally supports up to 256 fields. Note that the original contents of the fields are not stored in the Manticore index. The text that you send to Manticore gets processed, and a full-text index (a special data structure that enables quick searches for a keyword) gets built from that text. But the original text contents are then simply discarded. Manticore assumes that you store those contents elsewhere anyway. Moreover, it is impossible to fully reconstruct the original text, because the specific whitespace, capitalization, punctuation, etc will all be lost during indexing. It is theoretically possible to partially reconstruct a given document from the Manticore full-text index, but that would be a slow process (especially if the CRC dictionary is used, which does not even store the original keywords and works with their hashes instead). ### Attributes¶ Attributes are additional values associated with each document that can be used to perform additional filtering and sorting during search. It is often desired to additionally process full-text search results based not only on matching document ID and its rank, but on a number of other per-document values as well. For instance, one might need to sort news search results by date and then relevance, or search through products within specified price range, or limit blog search to posts made by selected users, or group results by month. To do that efficiently, Manticore allows to attach a number of additional attributes to each document, and store their values in the full-text index. It’s then possible to use stored values to filter, sort, or group full-text matches. Attributes, unlike the fields, are not full-text indexed. They are stored in the index, but it is not possible to search them as full-text, and attempting to do so results in an error. For example, it is impossible to use the extended matching mode expression @column 1 to match documents where column is 1, if column is an attribute, and this is still true even if the numeric digits are normally indexed. Attributes can be used for filtering, though, to restrict returned rows, as well as sorting or result grouping; it is entirely possible to sort results purely based on attributes, and ignore the search relevance tools. Additionally, attributes are returned from the search daemon, while the indexed text is not. A good example for attributes would be a forum posts table. Assume that only title and content fields need to be full-text searchable - but that sometimes it is also required to limit search to a certain author or a sub-forum (ie. search only those rows that have some specific values of author_id or forum_id columns in the SQL table); or to sort matches by post_date column; or to group matching posts by month of the post_date and calculate per-group match counts. This can be achieved by specifying all the mentioned columns (excluding title and content, that are full-text fields) as attributes, indexing them, and then using API calls to setup filtering, sorting, and grouping. Here as an example. #### Example sphinx.conf part:¶ ... sql_query = SELECT id, title, content, \ author_id, forum_id, post_date FROM my_forum_posts sql_attr_uint = author_id sql_attr_uint = forum_id sql_attr_timestamp = post_date ...  #### Example application code (in PHP):¶ // only search posts by author whose ID is 123$cl->SetFilter ( "author_id", array ( 123 ) );

// only search posts in sub-forums 1, 3 and 7
$cl->SetFilter ( "forum_id", array ( 1,3,7 ) ); // sort found posts by posting date in descending order$cl->SetSortMode ( SPH_SORT_ATTR_DESC, "post_date" );


Attributes are named. Attribute names are case insensitive. Attributes are not full-text indexed; they are stored in the index as is. Currently supported attribute types are:

• unsigned integers (1-bit to 32-bit wide);
• signed big integers (64-bit wide);
• UNIX timestamps;
• floating point values (32-bit, IEEE 754 single precision);
• strings;
• JSON;
• MVA, multi-value attributes (variable-length lists of 32-bit unsigned or 64-bit signed integers).

The complete set of per-document attribute values is sometimes referred to as docinfo. Docinfos are stored separately from the main full-text index data in .spa file.

A copy of .spa file (with all the attribute values for all the documents) is kept in RAM by searchd at all times (via mmap()). This is for performance reasons; random disk I/O may be too slow otherwise in most cases.

### MVA (multi-valued attributes)¶

MVAs, or multi-valued attributes, are an important special type of per-document attributes in Manticore. MVAs let you attach sets of numeric values to every document. That is useful to implement article tags, product categories, etc. Filtering and group-by (but not sorting) on MVA attributes is supported.

MVA values can either be unsigned 32-bit integers (UNSIGNED INTEGER) or signed 64-bit integers (BIGINT).

The set size is not limited, you can have an arbitrary number of values attached to each document as long as RAM permits (.spb file that contains the MVA values will be precached in RAM by searchd). The source data can be taken either from a separate query, or from a document field; see source type in sql_attr_multi. In the first case the query will have to return pairs of document ID and MVA values, in the second one the field will be parsed for integer values. There are absolutely no requirements as to incoming data order; the values will be automatically grouped by document ID (and internally sorted within the same ID) during indexing anyway.

When filtering, a document will match the filter on MVA attribute if any of the values satisfy the filtering condition. (Therefore, documents that pass through exclude filters will not contain any of the forbidden values.) When grouping by MVA attribute, a document will contribute to as many groups as there are different MVA values associated with that document. For instance, if the collection contains exactly 1 document having a ‘tag’ MVA with values 5, 7, and 11, grouping on ‘tag’ will produce 3 groups with ‘COUNT(*)‘equal to 1 and ‘GROUPBY()’ key values of 5, 7, and 11 respectively. Also note that grouping by MVA might lead to duplicate documents in the result set: because each document can participate in many groups, it can be chosen as the best one in in more than one group, leading to duplicate IDs. PHP API historically uses ordered hash on the document ID for the resulting rows; so you’ll also need to use SetArrayResult() in order to employ group-by on MVA with PHP API.

### Restrictions on the source data¶

There are a few different restrictions imposed on the source data which is going to be indexed by Manticore, of which the single most important one is:

ALL DOCUMENT IDS MUST BE UNIQUE SIGNED POSITIVE NON-ZERO 64-BIT INTEGER NUMBERS

If this requirement is not met, different bad things can happen. For instance, Manticore can crash with an internal assertion while indexing; or produce strange results when searching due to conflicting IDs. Also, a 1000-pound gorilla might eventually come out of your display and start throwing barrels at you. You’ve been warned.

### Charsets, case folding, translation tables, and replacement rules¶

When indexing some index, Manticore fetches documents from the specified sources, splits the text into words, and does case folding so that “Abc”, “ABC” and “abc” would be treated as the same word (or, to be pedantic, term).

To do that properly, Manticore needs to know

• what encoding is the source text in (and this encoding should always be UTF-8);
• what characters are letters and what are not;
• what letters should be folded to what letters.

This should be configured on a per-index basis using charset_table. option. charset_table specifies the table that maps letter characters to their case folded versions. The characters that are not in the table are considered to be non-letters and will be treated as word separators when indexing or searching through this index.

Default tables currently include English and Russian characters. Please do submit your tables for other languages!

You can also specify text pattern replacement rules. For example, given the rules

regexp_filter = \**(\d+)\" => \1 inch
regexp_filter = (BLUE|RED) => COLOR


the text ‘RED TUBE 5” LONG’ would be indexed as ‘COLOR TUBE 5 INCH LONG’, and ‘PLANK 2” x 4“‘as ‘PLANK 2 INCH x 4 INCH’. Rules are applied in the given order. Text in queries is also replaced; a search for”BLUE TUBE” would actually become a search for “COLOR TUBE”. Note that Manticore must be built with the –with-re2 option to use this feature.

### Data sources¶

The data to be indexed can generally come from very different sources: SQL databases, plain text files, HTML files, mailboxes, and so on. From Manticore point of view, the data it indexes is a set of structured documents, each of which has the same set of fields and attributes. This is similar to SQL, where each row would correspond to a document, and each column to either a field or an attribute.

Depending on what source Manticore should get the data from, different code is required to fetch the data and prepare it for indexing. This code is called data source driver (or simply driver or data source for brevity).

At the time of this writing, there are built-in drivers for MySQL, PostgreSQL, MS SQL (on Windows), and ODBC. There is also a generic driver called xmlpipe2, which runs a specified command and reads the data from its stdout. See xmlpipe2 data source section for the format description. tsvpipe (Tab Separated Values) and csvpipe (Comma Separated Values) data source also available and described in TSV/CSV data source.

There can be as many sources per index as necessary. They will be sequentially processed in the very same order which was specified in index definition. All the documents coming from those sources will be merged as if they were coming from a single source.

### SQL data sources (MySQL, PostgreSQL)¶

With all the SQL drivers, indexing generally works as follows.

• connection to the database is established;
• pre-query (see sql_query_pre) is executed to perform any necessary initial setup, such as setting per-connection encoding with MySQL;
• main query (see sql_query) is executed and the rows it returns are indexed;
• post-query (see sql_query_post) is executed to perform any necessary cleanup;
• connection to the database is closed;
• indexer does the sorting phase (to be pedantic, index-type specific post-processing);
• connection to the database is established again;
• post-index query (see sql_query_post_index) is executed to perform any necessary final cleanup;
• connection to the database is closed again.

Most options, such as database user/host/password, are straightforward. However, there are a few subtle things, which are discussed in more detail here.

#### Ranged queries¶

Main query, which needs to fetch all the documents, can impose a read lock on the whole table and stall the concurrent queries (eg. INSERTs to MyISAM table), waste a lot of memory for result set, etc. To avoid this, Manticore supports so-called ranged queries. With ranged queries, Manticore first fetches min and max document IDs from the table, and then substitutes different ID intervals into main query text and runs the modified query to fetch another chunk of documents. Here’s an example.

Example 3.1. Ranged query usage example

# in sphinx.conf

sql_query_range = SELECT MIN(id),MAX(id) FROM documents
sql_range_step = 1000
sql_query = SELECT * FROM documents WHERE id>=$start AND id<=$end


If the table contains document IDs from 1 to, say, 2345, then sql_query would be run three times:

1. with $start replaced with 1 and $end replaced with 1000;
2. with $start replaced with 1001 and $end replaced with 2000;
3. with $start replaced with 2001 and $end replaced with 2345.

Obviously, that’s not much of a difference for 2000-row table, but when it comes to indexing 10-million-row MyISAM table, ranged queries might be of some help.

#### sql_query_post vs. sql_query_post_index¶

The difference between post-query and post-index query is in that post-query is run immediately when Manticore received all the documents, but further indexing may still fail for some other reason. On the contrary, by the time the post-index query gets executed, it is guaranteed that the indexing was successful. Database connection is dropped and re-established because sorting phase can be very lengthy and would just timeout otherwise.

### xmlpipe2 data source¶

xmlpipe2 lets you pass arbitrary full-text and attribute data to Manticore in yet another custom XML format. It also allows to specify the schema (ie. the set of fields and attributes) either in the XML stream itself, or in the source settings.

When indexing xmlpipe2 source, indexer runs the given command, opens a pipe to its stdout, and expects well-formed XML stream. Here’s sample stream data:

Example 3.2. xmlpipe2 document stream

<?xml version="1.0" encoding="utf-8"?>
<sphinx:docset>

<sphinx:schema>
<sphinx:field name="subject"/>
<sphinx:field name="content"/>
<sphinx:attr name="published" type="timestamp"/>
<sphinx:attr name="author_id" type="int" bits="16" default="1"/>
</sphinx:schema>

<sphinx:document id="1234">
must be handled properly by xml parser lib]]></content>
<published>1012325463</published>
<subject>note how field/attr tags can be
in <b> class="red">randomized</b> order</subject>
<misc>some undeclared element</misc>
</sphinx:document>

<sphinx:document id="1235">
<subject>another subject</subject>
<content>here comes another document, and i am given to understand,
that in-document field order must not matter, sir</content>
<published>1012325467</published>
</sphinx:document>

<!-- ... even more sphinx:document entries here ... -->

<sphinx:killlist>
<id>1234</id>
<id>4567</id>
</sphinx:killlist>

</sphinx:docset>


Arbitrary fields and attributes are allowed. They also can occur in the stream in arbitrary order within each document; the order is ignored. There is a restriction on maximum field length; fields longer than 2 MB will be truncated to 2 MB (this limit can be changed in the source).

The schema, ie. complete fields and attributes list, must be declared before any document could be parsed. This can be done either in the configuration file using xmlpipe_field and xmlpipe_attr_XXX settings, or right in the stream using <sphinx:schema> element. <sphinx:schema> is optional. It is only allowed to occur as the very first sub-element in <sphinx:docset>. If there is no in-stream schema definition, settings from the configuration file will be used. Otherwise, stream settings take precedence.

Unknown tags (which were not declared neither as fields nor as attributes) will be ignored with a warning. In the example above, <misc> will be ignored. All embedded tags and their attributes (such as ** in <subject> in the example above) will be silently ignored.

Support for incoming stream encodings depends on whether iconv is installed on the system. xmlpipe2 is parsed using libexpat parser that understands US-ASCII, ISO-8859-1, UTF-8 and a few UTF-16 variants natively. Manticore configure script will also check for libiconv presence, and utilize it to handle other encodings. libexpat also enforces the requirement to use UTF-8 charset on Manticore side, because the parsed data it returns is always in UTF-8.

XML elements (tags) recognized by xmlpipe2 (and their attributes where applicable) are:

• sphinx:docset
• Mandatory top-level element, denotes and contains xmlpipe2 document set.
• sphinx:schema
• Optional element, must either occur as the very first child of sphinx:docset, or never occur at all. Declares the document schema. Contains field and attribute declarations. If present, overrides per-source settings from the configuration file.
• sphinx:field
• Optional element, child of sphinx:schema. Declares a full-text field. Known attributes are:
• “name”, specifies the XML element name that will be treated as a full-text field in the subsequent documents.
• “attr”, specifies whether to also index this field as a string. Possible value is “string”.
• sphinx:attr
• Optional element, child of sphinx:schema. Declares an attribute. Known attributes are:
• “name”, specifies the element name that should be treated as an attribute in the subsequent documents.
• “type”, specifies the attribute type. Possible values are “int”, “bigint”, “timestamp”, “bool”, “float”, “multi” and “json”.
• “bits”, specifies the bit size for “int” attribute type. Valid values are 1 to 32.
• “default”, specifies the default value for this attribute that should be used if the attribute’s element is not present in the document.
• sphinx:document
• Mandatory element, must be a child of sphinx:docset. Contains arbitrary other elements with field and attribute values to be indexed, as declared either using sphinx:field and sphinx:attr elements or in the configuration file. The only known attribute is “id” that must contain the unique integer document ID.
• sphinx:killlist
• Optional element, child of sphinx:docset. Contains a number of “id” elements whose contents are document IDs to be put into a kill-list for this index.

### TSV/CSV data source¶

This is the simplest way to pass data to the indexer. It was created due to xmlpipe2 limitations. Namely, indexer must map each attribute and field tag in XML file to corresponding schema element. This mapping requires some time. And time increases with increasing the number of fields and attributes in schema. There is no such issue in tsvpipe because each field and attribute is a particular column in TSV file. So, in some cases tsvpipe could work slightly faster than xmlpipe2.

The first column in TSVCSV file must be a document ID. The rest ones must mirror the declaration of fields and attributes in schema definition.

The difference between tsvpipe and csvpipe is delimiter and quoting rules. tsvpipe has tab character as hardcoded delimiter and has no quoting rules. csvpipe has option csvpipe_delimiter for delimiter with default value ‘,’ and also has quoting rules, such as:

• any field may be quoted
• fields containing a line-break, double-quote or commas should be quoted
• a double quote character in a field must be represented by two double quote characters

tsvpipe and csvpipe have same field and attrribute declaration derectives as xmlpipe.

tsvpipe declarations:

csvpipe declarations:

source tsv_test
{
type = tsvpipe
tsvpipe_command = cat /tmp/rock_bands.tsv
tsvpipe_field = name
tsvpipe_attr_multi = genre_tags
}

1   Led Zeppelin    35,23,16
2   Deep Purple 35,92
3   Frank Zappa 35,23,16,92,33,24


There are two major approaches to maintaining the full-text index contents up to date. Note, however, that both these approaches deal with the task of full-text data updates, and not attribute updates (which are already supported, refer to UpdateAttributes API call description for details.)

First, you can use disk-based indexes, partition them manually, and only rebuild the smaller partitions (so-called “deltas”) frequently. By minimizing the rebuild size, you can reduce the average indexing lag to something as low as 30-60 seconds. On huge collections it actually might be the most efficient one. Refer to Delta index updates for details.

Second, using real-time indexes (RT indexes for short) that on-the-fly updates of the full-text data. Updates on a RT index can appear in the search results in 1-2 milliseconds, ie. 0.001-0.002 seconds. However, RT index are less efficient for bulk indexing huge amounts of data. Refer to Real-time indexes for details.

There’s a frequent situation when the total dataset is too big to be reindexed from scratch often, but the amount of new records is rather small. Example: a forum with a 1,000,000 archived posts, but only 1,000 new posts per day.

In this case, “live” (almost real time) index updates could be implemented using so called “main+delta” scheme.

The idea is to set up two sources and two indexes, with one “main” index for the data which only changes rarely (if ever), and one “delta” for the new documents. In the example above, 1,000,000 archived posts would go to the main index, and newly inserted 1,000 posts/day would go to the delta index. Delta index could then be reindexed very frequently, and the documents can be made available to search in a matter of minutes.

Specifying which documents should go to what index and reindexing main index could also be made fully automatic. One option would be to make a counter table which would track the ID which would split the documents, and update it whenever the main index is reindexed.

Example 3.3. Fully automated live updates

# in MySQL
CREATE TABLE sph_counter
(
counter_id INTEGER PRIMARY KEY NOT NULL,
max_doc_id INTEGER NOT NULL
);

# in sphinx.conf
source main
{
# ...
sql_query_pre = SET NAMES utf8
sql_query_pre = REPLACE INTO sph_counter SELECT 1, MAX(id) FROM documents
sql_query = SELECT id, title, body FROM documents \
WHERE id<=( SELECT max_doc_id FROM sph_counter WHERE counter_id=1 )
}

source delta : main
{
sql_query_pre = SET NAMES utf8
sql_query = SELECT id, title, body FROM documents \
WHERE id>( SELECT max_doc_id FROM sph_counter WHERE counter_id=1 )
}

index main
{
source = main
path = /path/to/main
# ... all the other settings
}

# note how all other settings are copied from main,
# but source and path are overridden (they MUST be)
index delta : main
{
source = delta
path = /path/to/delta
}


Note how we’re overriding sql_query_pre in the delta source. We need to explicitly have that override. Otherwise REPLACE query would be run when indexing delta source too, effectively nullifying it. However, when we issue the directive in the inherited source for the first time, it removes all inherited values, so the encoding setup is also lost. So sql_query_pre in the delta can not just be empty; and we need to issue the encoding setup query explicitly once again.

### Index merging¶

Merging two existing indexes can be more efficient than indexing the data from scratch, and desired in some cases (such as merging ‘main’ and ‘delta’ indexes instead of simply reindexing ‘main’ in ‘main+delta’ partitioning scheme). So indexer has an option to do that. Merging the indexes is normally faster than reindexing but still not instant on huge indexes. Basically, it will need to read the contents of both indexes once and write the result once. Merging 100 GB and 1 GB index, for example, will result in 202 GB of IO (but that’s still likely less than the indexing from scratch requires).

The basic command syntax is as follows:

indexer --merge DSTINDEX SRCINDEX [--rotate]


Only the DSTINDEX index will be affected: the contents of SRCINDEX will be merged into it. --rotate switch will be required if DSTINDEX is already being served by searchd. The initially devised usage pattern is to merge a smaller update from SRCINDEX into DSTINDEX. Thus, when merging the attributes, values from SRCINDEX will win if duplicate document IDs are encountered. Note, however, that the “old” keywords will not be automatically removed in such cases. For example, if there’s a keyword “old” associated with document 123 in DSTINDEX, and a keyword “new” associated with it in SRCINDEX, document 123 will be found by both keywords after the merge. You can supply an explicit condition to remove documents from DSTINDEX to mitigate that; the relevant switch is --merge-dst-range:

indexer --merge main delta --merge-dst-range deleted 0 0


This switch lets you apply filters to the destination index along with merging. There can be several filters; all of their conditions must be met in order to include the document in the resulting merged index. In the example above, the filter passes only those records where ‘deleted’ is 0, eliminating all records that were flagged as deleted (for instance, using UpdateAttributes() call).

## Real-time indexes¶

Real-time indexes (or RT indexes for brevity) are a backend that lets you insert, update, or delete documents (rows) on the fly. While querying of RT indexes is possible using any of the SphinxAPI, SphinxQL, or SphinxSE, updating them is only possible via SphinxQL at the moment. Full SphinxQL reference is available in SphinxQL reference.

### RT indexes overview¶

RT indexes should be declared in sphinx.conf, just as every other index type. Notable differences from the regular, disk-based indexes are that a) data sources are not required and ignored, and b) you should explicitly enumerate all the text fields, not just attributes. Here’s an example:

Example 4.1. RT index declaration

index rt
{
type = rt
path = /usr/local/sphinx/data/rt
rt_field = title
rt_field = content
rt_attr_uint = gid
}


RT index can be accessed using MySQL protocol. INSERT, REPLACE, DELETE, and SELECT statements against RT index are supported. For instance, this is an example session with the sample index above:

$mysql -h 127.0.0.1 -P 9306 Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1 Server version: 1.10-dev (r2153) Type 'help;' or '\h' for help. Type '\c' to clear the buffer. mysql> INSERT INTO rt VALUES ( 1, 'first record', 'test one', 123 ); Query OK, 1 row affected (0.05 sec) mysql> INSERT INTO rt VALUES ( 2, 'second record', 'test two', 234 ); Query OK, 1 row affected (0.00 sec) mysql> SELECT * FROM rt; +------+--------+------+ | id | weight | gid | +------+--------+------+ | 1 | 1 | 123 | | 2 | 1 | 234 | +------+--------+------+ 2 rows in set (0.02 sec) mysql> SELECT * FROM rt WHERE MATCH('test'); +------+--------+------+ | id | weight | gid | +------+--------+------+ | 1 | 1643 | 123 | | 2 | 1643 | 234 | +------+--------+------+ 2 rows in set (0.01 sec) mysql> SELECT * FROM rt WHERE MATCH('@title test'); Empty set (0.00 sec)  Both partial and batch INSERT syntaxes are supported, ie. you can specify a subset of columns, and insert several rows at a time. Deletions are also possible using DELETE statement; the only currently supported syntax is DELETE FROM <index> WHERE id=<id>. REPLACE is also supported, enabling you to implement updates. mysql> INSERT INTO rt ( id, title ) VALUES ( 3, 'third row' ), ( 4, 'fourth entry' ); Query OK, 2 rows affected (0.01 sec) mysql> SELECT * FROM rt; +------+--------+------+ | id | weight | gid | +------+--------+------+ | 1 | 1 | 123 | | 2 | 1 | 234 | | 3 | 1 | 0 | | 4 | 1 | 0 | +------+--------+------+ 4 rows in set (0.00 sec) mysql> DELETE FROM rt WHERE id=2; Query OK, 0 rows affected (0.00 sec) mysql> SELECT * FROM rt WHERE MATCH('test'); +------+--------+------+ | id | weight | gid | +------+--------+------+ | 1 | 1500 | 123 | +------+--------+------+ 1 row in set (0.00 sec) mysql> INSERT INTO rt VALUES ( 1, 'first record on steroids', 'test one', 123 ); ERROR 1064 (42000): duplicate id '1' mysql> REPLACE INTO rt VALUES ( 1, 'first record on steroids', 'test one', 123 ); Query OK, 1 row affected (0.01 sec) mysql> SELECT * FROM rt WHERE MATCH('steroids'); +------+--------+------+ | id | weight | gid | +------+--------+------+ | 1 | 1500 | 123 | +------+--------+------+ 1 row in set (0.01 sec)  Data stored in RT index should survive clean shutdown. When binary logging is enabled, it should also survive crash and/or dirty shutdown, and recover on subsequent startup. ### Known caveats with RT indexes¶ RT indexes are currently quality feature, but there are still a few known usage quirks. Those quirks are listed in this section. • Default conservative RAM chunk limit (rt_mem_limit) of 32M can lead to poor performance on bigger indexes, you should raise it to 256..1024M if you’re planning to index gigabytes. • High DELETE/REPLACE rate can lead to kill-list fragmentation and impact searching performance. • No transaction size limits are currently imposed; too many concurrent INSERT/REPLACE transactions might therefore consume a lot of RAM. • In case of a damaged binlog, recovery will stop on the first damaged transaction, even though it’s technically possible to keep looking further for subsequent undamaged transactions, and recover those. This mid-file damage case (due to flaky HDD/CDD/tape?) is supposed to be extremely rare, though. • Multiple INSERTs grouped in a single transaction perform better than equivalent single-row transactions and are recommended for batch loading of data. ### RT index internals¶ RT index is internally chunked. It keeps a so-called RAM chunk that stores all the most recent changes. RAM chunk memory usage is rather strictly limited with per-index rt_mem_limit directive. Once RAM chunk grows over this limit, a new disk chunk is created from its data, and RAM chunk is reset. Thus, while most changes on the RT index will be performed in RAM only and complete instantly (in milliseconds), those changes that overflow the RAM chunk will stall for the duration of disk chunk creation (a few seconds). Manticore uses double-buffering to avoid INSERT stalls. When data is being dumped to disk, the second buffer is used, so further INSERTs won’t be delayed. The second buffer is defined to be 10% the size of the standard buffer, rt_mem_limit, but future versions of Manticore may allow configuring this further. Disk chunks are, in fact, just regular disk-based indexes. But they’re a part of an RT index and automatically managed by it, so you need not configure nor manage them manually. Because a new disk chunk is created every time RT chunk overflows the limit, and because in-memory chunk format is close to on-disk format, the disk chunks will be approximately rt_mem_limit bytes in size each. Generally, it is better to set the limit bigger, to minimize both the frequency of flushes, and the index fragmentation (number of disk chunks). For instance, on a dedicated search server that handles a big RT index, it can be advised to set rt_mem_limit to 1-2 GB. A global limit on all indexes is also planned, but not yet implemented. Disk chunk full-text index data can not be actually modified, so the full-text field changes (ie. row deletions and updates) suppress a previous row version from a disk chunk using a kill-list, but do not actually physically purge the data. Therefore, on workloads with high full-text updates ratio index might eventually get polluted by these previous row versions, and searching performance would degrade. Physical index purging that would improve the performance may be performed with OPTIMIZE command. Data in RAM chunk gets saved to disk on clean daemon shutdown, and then loaded back on startup. However, on daemon or server crash, updates from RAM chunk might be lost. To prevent that, binary logging of transactions can be used; see the section called :ref:binary_logging for details. Full-text changes in RT index are transactional. They are stored in a per-thread accumulator until COMMIT, then applied at once. Bigger batches per single COMMIT should result in faster indexing. ### Binary logging¶ Binary logs are essentially a recovery mechanism. With binary logs enabled, searchd writes every given transaction to the binlog file, and uses that for recovery after an unclean shutdown. On clean shutdown, RAM chunks are saved to disk, and then all the binlog files are unlinked. During normal operation, a new binlog file will be opened every time when binlog_max_log_size limit is reached. Older, already closed binlog files are kept until all of the transactions stored in them (from all indexes) are flushed as a disk chunk. Setting the limit to 0 pretty much prevents binlog from being unlinked at all while searchd is running; however, it will still be unlinked on clean shutdown. (binlog_max_log_size defaults to 0.) There are 3 different binlog flushing strategies, controlled by binlog_flush directive which takes the values of 0, 1, or 2. 0 means to flush the log to OS and sync it to disk every second; 1 means flush and sync every transaction; and 2 (the default mode) means flush every transaction but sync every second. Sync is relatively slow because it has to perform physical disk writes, so mode 1 is the safest (every committed transaction is guaranteed to be written on disk) but the slowest. Flushing log to OS prevents from data loss on searchd crashes but not system crashes. Mode 2 is the default. On recovery after an unclean shutdown, binlogs are replayed and all logged transactions since the last good on-disk state are restored. Transactions are checksummed so in case of binlog file corruption garbage data will not be replayed; such a broken transaction will be detected and, currently, will stop replay. Transactions also start with a magic marker and timestamped, so in case of binlog damage in the middle of the file, it’s technically possible to skip broken transactions and keep replaying from the next good one, and/or it’s possible to replay transactions until a given timestamp (point-in-time recovery), but none of that is implemented yet. One unwanted side effect of binlogs is that actively updating a small RT index that fully fits into a RAM chunk part will lead to an ever-growing binlog that can never be unlinked until clean shutdown. Binlogs are essentially append-only deltas against the last known good saved state on disk, and unless RAM chunk gets saved, they can not be unlinked. An ever-growing binlog is not very good for disk use and crash recovery time. To avoid this, you can configure searchd to perform a periodic RAM chunk flush to fix that problem using a rt_flush_period directive. With periodic flushes enabled, searchd will keep a separate thread, checking whether RT indexes RAM chunks need to be written back to disk. Once that happens, the respective binlogs can be (and are) safely unlinked. Note that rt_flush_period only controls the frequency at which the checks happen. There are no guarantees that the particular RAM chunk will get saved. For instance, it does not make sense to regularly re-save a huge RAM chunk that only gets a few rows worth of updates. The search daemon determine whether to actually perform the flush with a few heuristics. ## Searching¶ ### Matching modes¶ So-called matching modes are a legacy feature that used to provide (very) limited query syntax and ranking support. Currently, they are deprecated in favor of full-text query language and so-called Available built-in rankers. It is thus strongly recommended to use SPH_MATCH_EXTENDED and proper query syntax rather than any other legacy mode. All those other modes are actually internally converted to extended syntax anyway. SphinxAPI still defaults to SPH_MATCH_ALL but that is for compatibility reasons only. There are the following matching modes available: • SPH_MATCH_ALL, matches all query words; • SPH_MATCH_ANY, matches any of the query words; • SPH_MATCH_PHRASE, matches query as a phrase, requiring perfect match; • SPH_MATCH_BOOLEAN, matches query as a boolean expression (see Boolean query syntax); • SPH_MATCH_EXTENDED, matches query as an expression in Manticore internal query language (see Extended query syntax); • SPH_MATCH_EXTENDED2, an alias for SPH_MATCH_EXTENDED (default mode); • SPH_MATCH_FULLSCAN, matches query, forcibly using the “full scan” mode as below. NB, any query terms will be ignored, such that filters, filter-ranges and grouping will still be applied, but no text-matching. The SPH_MATCH_FULLSCAN mode will be automatically activated in place of the specified matching mode when the query string is empty (ie. its length is zero). In full scan mode, all the indexed documents will be considered as matching. Such queries will still apply filters, sorting, and group by, but will not perform any full-text searching. This can be useful to unify full-text and non-full-text searching code, or to offload SQL server (there are cases when Manticore scans will perform better than analogous MySQL queries). An example of using the full scan mode might be to find posts in a forum. By selecting the forum’s user ID via SetFilter() but not actually providing any search text, Manticore will match every document (i.e. every post) where SetFilter() would match - in this case providing every post from that user. By default this will be ordered by relevancy, followed by Manticore document ID in ascending order (earliest first). ### Boolean query syntax¶ Boolean queries allow the following special operators to be used: • operator OR: hello | world  • operator NOT: hello -world hello !world  • grouping: ( hello world )  Here’s an example query which uses all these operators: Example 5.1. Boolean query example ( cat -dog ) | ( cat -mouse)  There always is implicit AND operator, so “hello world” query actually means “hello & world”. OR operator precedence is higher than AND, so “looking for cat | dog | mouse” means “looking for ( cat | dog | mouse )” and not “(looking for cat) | dog | mouse”. Queries may be automatically optimized if OPTION boolean_simplify=1 is specified. Some transformations performed by this optimization include: • Excess brackets: ((A | B) | C) becomes ( A | B | C ); ((A B) C) becomes ( A B C ) • Excess AND NOT: ((A !N1) !N2) becomes (A !(N1 | N2)) • Common NOT: ((A !N) | (B !N)) becomes ((A|B) !N) • Common Compound NOT: ((A !(N AA)) | (B !(N BB))) becomes (((A|B) !N) | (A !AA) | (B !BB)) if the cost of evaluating N is greater than the added together costs of evaluating A and B • Common subterm: ((A (N | AA)) | (B (N | BB))) becomes (((A|B) N) | (A AA) | (B BB)) if the cost of evaluating N is greater than the added together costs of evaluating A and B • Common keywords: (A | “A B”~N) becomes A; (“A B” | “A B C”) becomes “A B”; (“A B”~N | “A B C”~N) becomes (“A B”~N) • Common phrase: (“X A B” | “Y A B”) becomes ((“X|Y”) “A B”) • Common AND NOT: ((A !X) | (A !Y) | (A !Z)) becomes (A !(X Y Z)) • Common OR NOT: ((A !(N | N1)) | (B !(N | N2))) becomes (( (A !N1) | (B !N2) ) !N) Note that optimizing the queries consumes CPU time, so for simple queries -or for hand-optimized queries- you’ll do better with the default boolean_simplify=0 value. Simplifications are often better for complex queries, or algorithmically generated queries. Queries like “-dog”, which implicitly include all documents from the collection, can not be evaluated. This is both for technical and performance reasons. Technically, Manticore does not always keep a list of all IDs. Performance-wise, when the collection is huge (ie. 10-100M documents), evaluating such queries could take very long. ### Escaping characters in search queries¶ A list of characters which should be escaped when they are searched for in SphinxQL fulltext search queries: ! "$    '    (    )    -    /    <    @    \    ^    |    ~


Use backslash to escape the single quote character ' like in the following example.

SphinxQL query to search for occurences of "l'italiano" :

SELECT * FROM your_index WHERE MATCH(‘l\'italiano')


Use double backslash to escape all other characters.

For example, here is a query to search for occurences of "r&b" or "(official video)" :

SELECT * FROM your_index WHERE MATCH('r\\&b | \$$official video\$$')


Pay attention that in order to escape backslash character you should use "\\\\\\\\" syntax.

E.g., to find occurences of "\\ABC", use:

SELECT * FROM your_index WHERE MATCH('\\\\ABC')


Also, if you run your queries using some programming language don’t forget about a mysql escaping function (e.g., mysqli_real_escape_string in PHP or conn.escape_string in Python) to escape these characters the same way as described above.

### Extended query syntax¶

The following special operators and modifiers can be used when using the extended matching mode:

• operator OR:
hello | world

• operator MAYBE:
hello MAYBE world

• operator NOT:
hello -world
hello !world

• field search operator:
@title hello @body world

• field position limit modifier:
@body[50] hello

• multiple-field search operator:
@(title,body) hello world

• ignore field search operator (will ignore any matches of ‘hello world’ from field ‘title’):
@!title hello world

• ignore multiple-field search operator (if we have fields title, subject and body then @!(title) is equivalent to @(subject,body)):
@!(title,body) hello world

• all-field search operator:
@* hello

• phrase search operator:
"hello world"

• proximity search operator:
"hello world"~10

• quorum matching operator:
"the world is a wonderful place"/3

• strict order operator (aka operator “before”):
aaa << bbb << ccc

• exact form modifier:
raining =cats and =dogs

• field-start and field-end modifier:
^hello world$ • keyword IDF boost modifier: boosted^1.234 boostedfieldend$^1.234

• NEAR, generalized proximity operator:
hello NEAR/3 world NEAR/4 "my test"

• SENTENCE operator:
all SENTENCE words SENTENCE "in one sentence"

• PARAGRAPH operator:
"Bill Gates" PARAGRAPH "Steve Jobs"

• ZONE limit operator:
    ZONE:(h3,h4)

only in these titles

• ZONESPAN limit operator:
    ZONESPAN:(h2)

only in a (single) title

• NOTNEAR, negative assertion operator:
Church NOTNEAR/3 street


Here’s an example query that uses some of these operators:

Example 5.2. Extended matching mode: query example

"hello world" @title "example program"~5 @body python -(php|perl) @* code


The full meaning of this search is:

• Find the words ‘hello’ and ‘world’ adjacently in any field in a document;
• Additionally, the same document must also contain the words ‘example’ and ‘program’ in the title field, with up to, but not including, 5 words between the words in question; (E.g. “example PHP program” would be matched however “example script to introduce outside data into the correct context for your program” would not because two terms have 5 or more words between them)
• Additionally, the same document must contain the word ‘python’ in the body field, but not contain either ‘php’ or ‘perl’;
• Additionally, the same document must contain the word ‘code’ in any field.

There always is implicit AND operator, so “hello world” means that both “hello” and “world” must be present in matching document.

OR operator precedence is higher than AND, so “looking for cat | dog | mouse” means “looking for ( cat | dog | mouse )” and not “(looking for cat) | dog | mouse”.

Field limit operator limits subsequent searching to a given field. Normally, query will fail with an error message if given field name does not exist in the searched index. However, that can be suppressed by specifying “@@relaxed” option at the very beginning of the query:

@@relaxed @nosuchfield my query


This can be helpful when searching through heterogeneous indexes with different schemas.

Field position limit additionally restricts the searching to first N position within given field (or fields). For example, “@body [50] hello” will not match the documents where the keyword ‘hello’ occurs at position 51 and below in the body.

Proximity distance is specified in words, adjusted for word count, and applies to all words within quotes. For instance, “cat dog mouse”~5 query means that there must be less than 8-word span which contains all 3 words, ie. “CAT aaa bbb ccc DOG eee fff MOUSE” document will not match this query, because this span is exactly 8 words long.

Quorum matching operator introduces a kind of fuzzy matching. It will only match those documents that pass a given threshold of given words. The example above (“the world is a wonderful place”/3) will match all documents that have at least 3 of the 6 specified words. Operator is limited to 255 keywords. Instead of an absolute number, you can also specify a number between 0.0 and 1.0 (standing for 0% and 100%), and Manticore will match only documents with at least the specified percentage of given words. The same example above could also have been written “the world is a wonderful place”/0.5 and it would match documents with at least 50% of the 6 words.

Strict order operator (aka operator “before”) will match the document only if its argument keywords occur in the document exactly in the query order. For instance, “black << cat” query (without quotes) will match the document “black and white cat” but not the “that cat was black” document. Order operator has the lowest priority. It can be applied both to just keywords and more complex expressions, ie. this is a valid query:

(bag of words) << "exact phrase" << red|green|blue


Exact form keyword modifier will match the document only if the keyword occurred in exactly the specified form. The default behavior is to match the document if the stemmed keyword matches. For instance, “runs” query will match both the document that contains “runs” and the document that contains “running”, because both forms stem to just “run” - while “=runs” query will only match the first document. Exact form operator requires index_exact_words option to be enabled. This is a modifier that affects the keyword and thus can be used within operators such as phrase, proximity, and quorum operators. It is possible to apply an exact form modifier to the phrase operator. It’s really just syntax sugar - it adds an exact form modifier to all terms contained within the phrase.

="exact phrase"


Field-start and field-end keyword modifiers will make the keyword match only if it occurred at the very start or the very end of a fulltext field, respectively. For instance, the query “^hello world$” (with quotes and thus combining phrase operator and start/end modifiers) will only match documents that contain at least one field that has exactly these two keywords. Arbitrarily nested brackets and negations are allowed. However, the query must be possible to compute without involving an implicit list of all documents: // correct query aaa -(bbb -(ccc ddd)) // queries that are non-computable -aaa aaa | -bbb  The phrase search operator may include a ‘match any term’ modifier. Terms within the phrase operator are position significant. When the ‘match any term’ modifier is implemented, the position of the subsequent terms from that phrase query will be shifted. Therefore, ‘match any’ has no impact on search performance. "exact * phrase * * for terms"  NEAR operator is a generalized version of a proximity operator. The syntax is NEAR/N, it is case-sensitive, and no spaces are allowed between the NEAR keyword, the slash sign, and the distance value. The original proximity operator only worked on sets of keywords. NEAR is more generic and can accept arbitrary subexpressions as its two arguments, matching the document when both subexpressions are found within N words of each other, no matter in which order. NEAR is left associative and has the same (lowest) precedence as BEFORE. You should also note how a (one NEAR/7 two NEAR/7 three) query using NEAR is not really equivalent to a ("one two three"~7) one using keyword proximity operator. The difference here is that the proximity operator allows for up to 6 non-matching words between all the 3 matching words, but the version with NEAR is less restrictive: it would allow for up to 6 words between ‘one’ and ‘two’ and then for up to 6 more between that two-word matching and a ‘three’ keyword. SENTENCE and PARAGRAPH operators matches the document when both its arguments are within the same sentence or the same paragraph of text, respectively. The arguments can be either keywords, or phrases, or the instances of the same operator. Here are a few examples: one SENTENCE two one SENTENCE "two three" one SENTENCE "two three" SENTENCE four  The order of the arguments within the sentence or paragraph does not matter. These operators only work on indexes built with index_sp (sentence and paragraph indexing feature) enabled, and revert to a mere AND otherwise. Refer to the index_sp directive documentation for the notes on what’s considered a sentence and a paragraph. ZONE limit operator is quite similar to field limit operator, but restricts matching to a given in-field zone or a list of zones. Note that the subsequent subexpressions are not required to match in a single contiguous span of a given zone, and may match in multiple spans. For instance, (ZONE:th hello world) query will match this example document: <th>Table 1\. Local awareness of Hello Kitty brand.</th> .. some table data goes here .. <th>Table 2\. World-wide brand awareness.</th>  ZONE operator affects the query until the next field or ZONE limit operator, or the closing parenthesis. It only works on the indexes built with zones support (see index_zones) and will be ignored otherwise. ZONESPAN limit operator is similar to the ZONE operator, but requires the match to occur in a single contiguous span. In the example above, (ZONESPAN:th hello world) would not match the document, since “hello” and “world” do not occur within the same span. MAYBE operator works much like | operator but doesn’t return documents which match only right subtree expression. NOTNEAR operator is a negative assertion. It matches the document when left argument exists and either there is no right argument in document or right argument is distance away from left matched argument’s end. The distance is specified in words. The syntax is NOTNEAR/N, it is case-sensitive, and no spaces are allowed between the NOTNEAR keyword, the slash sign, and the distance value. Both arguments of this operator might be terms or any operators or group of operators. ### Search results ranking¶ #### Ranking overview¶ Ranking (aka weighting) of the search results can be defined as a process of computing a so-called relevance (aka weight) for every given matched document with regards to a given query that matched it. So relevance is in the end just a number attached to every document that estimates how relevant the document is to the query. Search results can then be sorted based on this number and/or some additional parameters, so that the most sought after results would come up higher on the results page. There is no single standard one-size-fits-all way to rank any document in any scenario. Moreover, there can not ever be such a way, because relevance is subjective. As in, what seems relevant to you might not seem relevant to me. Hence, in general case it’s not just hard to compute, it’s theoretically impossible. So ranking in Manticore is configurable. It has a notion of a so-called ranker. A ranker can formally be defined as a function that takes document and query as its input and produces a relevance value as output. In layman’s terms, a ranker controls exactly how (using which specific algorithm) will Manticore assign weights to the document. Previously, this ranking function was rigidly bound to the matching mode. So in the legacy matching modes (that is, SPH_MATCH_ALL, SPH_MATCH_ANY, SPH_MATCH_PHRASE, and SPH_MATCH_BOOLEAN) you can not choose the ranker. You can only do that in the SPH_MATCH_EXTENDED mode. (Which is the only mode in SphinxQL and the suggested mode in SphinxAPI anyway.) To choose a non-default ranker you can either use SetRankingMode() with SphinxAPI, or OPTION ranker clause in SELECT statement when using SphinxQL. As a sidenote, legacy matching modes are internally implemented via the unified syntax anyway. When you use one of those modes, Manticore just internally adjusts the query and sets the associated ranker, then executes the query using the very same unified code path. #### Available built-in rankers¶ Manticore ships with a number of built-in rankers suited for different purposes. A number of them uses two factors, phrase proximity (aka LCS) and BM25. Phrase proximity works on the keyword positions, while BM25 works on the keyword frequencies. Basically, the better the degree of the phrase match between the document body and the query, the higher is the phrase proximity (it maxes out when the document contains the entire query as a verbatim quote). And BM25 is higher when the document contains more rare words. We’ll save the detailed discussion for later. Currently implemented rankers are: • SPH_RANK_PROXIMITY_BM25, the default ranking mode that uses and combines both phrase proximity and BM25 ranking. • SPH_RANK_BM25, statistical ranking mode which uses BM25 ranking only (similar to most other full-text engines). This mode is faster but may result in worse quality on queries which contain more than 1 keyword. • SPH_RANK_NONE, no ranking mode. This mode is obviously the fastest. A weight of 1 is assigned to all matches. This is sometimes called boolean searching that just matches the documents but does not rank them. • SPH_RANK_WORDCOUNT, ranking by the keyword occurrences count. This ranker computes the per-field keyword occurrence counts, then multiplies them by field weights, and sums the resulting values. • SPH_RANK_PROXIMITY returns raw phrase proximity value as a result. This mode is internally used to emulate SPH_MATCH_ALL queries. • SPH_RANK_MATCHANY returns rank as it was computed in SPH_MATCH_ANY mode earlier, and is internally used to emulate SPH_MATCH_ANY queries. • SPH_RANK_FIELDMASK returns a 32-bit mask with N-th bit corresponding to N-th fulltext field, numbering from 0. The bit will only be set when the respective field has any keyword occurrences satisfying the query. • SPH_RANK_SPH04 is generally based on the default SPH_RANK_PROXIMITY_BM25 ranker, but additionally boosts the matches when they occur in the very beginning or the very end of a text field. Thus, if a field equals the exact query, SPH04 should rank it higher than a field that contains the exact query but is not equal to it. (For instance, when the query is “Hyde Park”, a document entitled “Hyde Park” should be ranked higher than a one entitled “Hyde Park, London” or “The Hyde Park Cafe”.) • SPH_RANK_EXPR lets you specify the ranking formula in run time. It exposes a number of internal text factors and lets you define how the final weight should be computed from those factors. You can find more details about its syntax and a reference available factors in a subsection below. You should specify the SPH_RANK_ prefix and use capital letters only when using the SetRankingMode() call from the SphinxAPI. The API ports expose these as global constants. Using SphinxQL syntax, the prefix should be omitted and the ranker name is case insensitive. Example: // SphinxAPI$client->SetRankingMode ( SPH_RANK_SPH04 );

// SphinxQL
mysql_query ( "SELECT ... OPTION ranker=sph04" );

##### Legacy matching modes rankers¶

Legacy matching modes automatically select a ranker as follows:

• SPH_MATCH_ALL uses SPH_RANK_PROXIMITY ranker;
• SPH_MATCH_ANY uses SPH_RANK_MATCHANY ranker;
• SPH_MATCH_PHRASE uses SPH_RANK_PROXIMITY ranker;
• SPH_MATCH_BOOLEAN uses SPH_RANK_NONE ranker.

#### Quick summary of the ranking factors¶

Name Level Type Summary
max_lcs query int maximum possible LCS value for the current query
bm25 document int quick estimate of BM25(1.2, 0) without syntax support
bm25a(k1, b) document int precise BM25() value with configurable K1, B constants and syntax support
bm25f(k1, b, {field=weight, …}) document int precise BM25F() value with extra configurable field weights
query_word_count document int number of unique inclusive keywords in a query
doc_word_count document int number of unique keywords matched in the document
lcs field int Longest Common Subsequence between query and document, in words
user_weight field int user field weight
hit_count field int total number of keyword occurrences
word_count field int number of unique matched keywords
tf_idf field float sum(tf*idf) over matched keywords == sum(idf) over occurrences
min_hit_pos field int first matched occurrence position, in words, 1-based
min_best_span_pos field int first maximum LCS span position, in words, 1-based
exact_hit field bool whether query == field
min_idf field float min(idf) over matched keywords
max_idf field float max(idf) over matched keywords
sum_idf field float sum(idf) over matched keywords
exact_order field bool whether all query keywords were a) matched and b) in query order
min_gaps field int minimum number of gaps between the matched keywords over the matching spans
lccs field int Longest Common Contiguous Subsequence between query and document, in words
wlccs field float Weighted Longest Common Contiguous Subsequence, sum(idf) over contiguous keyword spans
atc field float Aggregate Term Closeness, log(1+sum(idf1*idf2*pow(distance, -1.75)) over the best pairs of keywords

#### Document-level ranking factors¶

A document-level factor is a numeric value computed by the ranking engine for every matched document with regards to the current query. So it differs from a plain document attribute in that the attribute do not depend on the full text query, while factors might. Those factors can be used anywhere in the ranking expression. Currently implemented document-level factors are:

• bm25 (integer), a document-level BM25 estimate (computed without keyword occurrence filtering).
• max_lcs (integer), a query-level maximum possible value that the sum(lcs*user_weight) expression can ever take. This can be useful for weight boost scaling. For instance, MATCHANY ranker formula uses this to guarantee that a full phrase match in any field ranks higher than any combination of partial matches in all fields.
• field_mask (integer), a document-level 32-bit mask of matched fields.
• query_word_count (integer), the number of unique keywords in a query, adjusted for a number of excluded keywords. For instance, both (one one one one) and (one !two) queries should assign a value of 1 to this factor, because there is just one unique non-excluded keyword.
• doc_word_count (integer), the number of unique keywords matched in the entire document.

#### Field-level ranking factors¶

A field-level factor is a numeric value computed by the ranking engine for every matched in-document text field with regards to the current query. As more than one field can be matched by a query, but the final weight needs to be a single integer value, these values need to be folded into a single one. To achieve that, field-level factors can only be used within a field aggregation function, they can not be used anywhere in the expression. For example, you can not use (lcs+bm25) as your ranking expression, as lcs takes multiple values (one in every matched field). You should use (sum(lcs)+bm25) instead, that expression sums lcs over all matching fields, and then adds bm25 to that per-field sum. Currently implemented field-level factors are:

• lcs (integer), the length of a maximum verbatim match between the document and the query, counted in words. LCS stands for Longest Common Subsequence (or Subset). Takes a minimum value of 1 when only stray keywords were matched in a field, and a maximum value of query keywords count when the entire query was matched in a field verbatim (in the exact query keywords order). For example, if the query is ‘hello world’ and the field contains these two words quoted from the query (that is, adjacent to each other, and exactly in the query order), lcs will be 2. For example, if the query is ‘hello world program’ and the field contains ‘hello world’, lcs will be 2. Note that any subset of the query keyword works, not just a subset of adjacent keywords. For example, if the query is ‘hello world program’ and the field contains ‘hello (test program)’, lcs will be 2 just as well, because both ‘hello’ and ‘program’ matched in the same respective positions as they were in the query. Finally, if the query is ‘hello world program’ and the field contains ‘hello world program’, lcs will be 3. (Hopefully that is unsurprising at this point.)

• user_weight (integer), the user specified per-field weight (refer to SetFieldWeights() in SphinxAPI and OPTION field_weights in SphinxQL respectively). The weights default to 1 if not specified explicitly.

• hit_count (integer), the number of keyword occurrences that matched in the field. Note that a single keyword may occur multiple times. For example, if ‘hello’ occurs 3 times in a field and ‘world’ occurs 5 times, hit_count will be 8.

• word_count (integer), the number of unique keywords matched in the field. For example, if ‘hello’ and ‘world’ occur anywhere in a field, word_count will be 2, irregardless of how many times do both keywords occur.

• tf_idf (float), the sum of TF/IDF over all the keywords matched in the field. IDF is the Inverse Document Frequency, a floating point value between 0 and 1 that describes how frequent is the keywords (basically, 0 for a keyword that occurs in every document indexed, and 1 for a unique keyword that occurs in just a single document). TF is the Term Frequency, the number of matched keyword occurrences in the field. As a side note, tf_idf is actually computed by summing IDF over all matched occurrences. That’s by construction equivalent to summing TF*IDF over all matched keywords.

• min_hit_pos (integer), the position of the first matched keyword occurrence, counted in words. Indexing begins from position 1.

• min_best_span_pos (integer), the position of the first maximum LCS occurrences span. For example, assume that our query was ‘hello world program’ and ‘hello world’ subphrase was matched twice in the field, in positions 13 and 21. Assume that ‘hello’ and ‘world’ additionally occurred elsewhere in the field, but never next to each other and thus never as a subphrase match. In that case, min_best_span_pos will be 13. Note how for the single keyword queries min_best_span_pos will always equal min_hit_pos.

• exact_hit (boolean), whether a query was an exact match of the entire current field. Used in the SPH04 ranker.

• min_idf, max_idf, and sum_idf (float). These factors respectively represent the min(idf), max(idf) and sum(idf) over all keywords that were matched in the field.

• exact_order (boolean). Whether all of the query keywords were matched in the field in the exact query order. For example, (microsoft office) query would yield exact_order=1 in a field with the following contents: (We use Microsoft software in our office.). However, the very same query in a (Our office is Microsoft free.) field would yield exact_order=0.

• min_gaps (integer), the minimum number of positional gaps between (just) the keywords matched in field. Always 0 when less than 2 keywords match; always greater or equal than 0 otherwise.

For example, with a [big wolf] query, [big bad wolf] field would yield min_gaps=1; [big bad hairy wolf] field would yield min_gaps=2; [the wolf was scary and big] field would yield min_gaps=3; etc. However, a field like [i heard a wolf howl] would yield min_gaps=0, because only one keyword would be matching in that field, and, naturally, there would be no gaps between the _matched_keywords.

Therefore, this is a rather low-level, “raw” factor that you would most likely want to adjust before actually using for ranking. Specific adjustments depend heavily on your data and the resulting formula, but here are a few ideas you can start with: (a) any min_gaps based boosts could be simply ignored when word_count<2; (b) non-trivial min_gaps values (i.e. when word_count>=2) could be clamped with a certain “worst case” constant while trivial values (i.e. when min_gaps=0 and word_count<2) could be replaced by that constant; (c) a transfer function like 1/(1+min_gaps) could be applied (so that better, smaller min_gaps values would maximize it and worse, bigger min_gaps values would fall off slowly); and so on.

• lccs (integer). Longest Common Contiguous Subsequence. A length of the longest subphrase that is common between the query and the document, computed in keywords.

LCCS factor is rather similar to LCS but more restrictive, in a sense. While LCS could be greater than 1 though no two query words are matched next to each other, LCCS would only get greater than 1 if there are exact, contiguous query subphrases in the document. For example, (one two three four five) query vs (one hundred three hundred five hundred) document would yield lcs=3, but lccs=1, because even though mutual dispositions of 3 keywords (one, three, five) match between the query and the document, no 2 matching positions are actually next to each other.

Note that LCCS still does not differentiate between the frequent and rare keywords; for that, see WLCCS.

• wlccs (float). Weighted Longest Common Contiguous Subsequence. A sum of IDFs of the keywords of the longest subphrase that is common between the query and the document.

WLCCS is computed very similarly to LCCS, but every “suitable” keyword occurrence increases it by the keyword IDF rather than just by 1 (which is the case with LCS and LCCS). That lets us rank sequences of more rare and important keywords higher than sequences of frequent keywords, even if the latter are longer. For example, a query (Zanzibar bed and breakfast) would yield lccs=1 for a (hotels of Zanzibar) document, but lccs=3 against (London bed and breakfast), even though “Zanzibar” is actually somewhat more rare than the entire “bed and breakfast” phrase. WLCCS factor alleviates that problem by using the keyword frequencies.

• atc (float). Aggregate Term Closeness. A proximity based measure that grows higher when the document contains more groups of more closely located and more important (rare) query keywords. WARNING: you should use ATC with OPTION idf=‘plain,tfidf_unnormalized’; otherwise you would get unexpected results.

ATC basically works as follows. For every keyword occurrence in the document, we compute the so called term closeness. For that, we examine all the other closest occurrences of all the query keywords (keyword itself included too) to the left and to the right of the subject occurrence, compute a distance dampening coefficient as k = pow(distance, -1.75) for those occurrences, and sum the dampened IDFs. Thus for every occurrence of every keyword, we get a “closeness” value that describes the “neighbors” of that occurrence. We then multiply those per-occurrence closenesses by their respective subject keyword IDF, sum them all, and finally, compute a logarithm of that sum.

Or in other words, we process the best (closest) matched keyword pairs in the document, and compute pairwise “closenesses” as the product of their IDFs scaled by the distance coefficient:

pair_tc = idf(pair_word1) * idf(pair_word2) * pow(pair_distance, -1.75)


We then sum such closenesses, and compute the final, log-dampened ATC value:

atc = log(1+sum(pair_tc))


Note that this final dampening logarithm is exactly the reason you should use OPTION idf=plain, because without it, the expression inside the log() could be negative.

Having closer keyword occurrences actually contributes much more to ATC than having more frequent keywords. Indeed, when the keywords are right next to each other, distance=1 and k=1; when there just one word in between them, distance=2 and k=0.297, with two words between, distance=3 and k=0.146, and so on. At the same time IDF attenuates somewhat slower. For example, in a 1 million document collection, the IDF values for keywords that match in 10, 100, and 1000 documents would be respectively 0.833, 0.667, and 0.500. So a keyword pair with two rather rare keywords that occur in just 10 documents each but with 2 other words in between would yield pair_tc = 0.101 and thus just barely outweigh a pair with a 100-doc and a 1000-doc keyword with 1 other word between them and pair_tc = 0.099. Moreover, a pair of two unique, 1-doc keywords with 3 words between them would get a pair_tc = 0.088 and lose to a pair of two 1000-doc keywords located right next to each other and yielding a pair_tc = 0.25. So, basically, while ATC does combine both keyword frequency and proximity, it is still somewhat favoring the proximity.

#### Ranking factor aggregation functions¶

A field aggregation function is a single argument function that takes an expression with field-level factors, iterates it over all the matched fields, and computes the final results. Currently implemented field aggregation functions are:

• sum, sums the argument expression over all matched fields. For instance, sum(1) should return a number of matched fields.
• top, returns the greatest value of the argument over all matched fields.

#### Formula expressions for all the built-in rankers¶

Most of the other rankers can actually be emulated with the expression based ranker. You just need to pass a proper expression. Such emulation is, of course, going to be slower than using the built-in, compiled ranker but still might be of interest if you want to fine-tune your ranking formula starting with one of the existing ones. Also, the formulas define the nitty gritty ranker details in a nicely readable fashion.

• SPH_RANK_PROXIMITY_BM25 = sum(lcs*user_weight)*1000+bm25
• SPH_RANK_BM25 = bm25
• SPH_RANK_NONE = 1
• SPH_RANK_WORDCOUNT = sum(hit_count*user_weight)
• SPH_RANK_PROXIMITY = sum(lcs*user_weight)
• SPH_RANK_MATCHANY = sum((word_count+(lcs-1)*max_lcs)*user_weight)
• SPH_RANK_SPH04 = sum((4*lcs+2*(min_hit_pos==1)+exact_hit)*user_weight)*1000+bm25

### Expressions, functions, and operators¶

Manticore lets you use arbitrary arithmetic expressions both via SphinxQL and SphinxAPI, involving attribute values, internal attributes (document ID and relevance weight), arithmetic operations, a number of built-in functions, and user-defined functions. This section documents the supported operators and functions. Here’s the complete reference list for quick access.

#### Operators¶

• Arithmetic operators: +, -, *, /, %, DIV, MOD

The standard arithmetic operators. Arithmetic calculations involving those can be performed in three different modes: (a) using single-precision, 32-bit IEEE 754 floating point values (the default), (**) using signed 32-bit integers, (c) using 64-bit signed integers. The expression parser will automatically switch to integer mode if there are no operations the result in a floating point value. Otherwise, it will use the default floating point mode. For instance, a+b will be computed using 32-bit integers if both arguments are 32-bit integers; or using 64-bit integers if both arguments are integers but one of them is 64-bit; or in floats otherwise. However, a/** or sqrt(a) will always be computed in floats, because these operations return a result of non-integer type. To avoid the first, you can either use IDIV(a,b) or a DIV b form. Also, a*b will not be automatically promoted to 64-bit when the arguments are 32-bit. To enforce 64-bit results, you can use BIGINT(). (But note that if there are non-integer operations, BIGINT() will simply be ignored.)

• Comparison operators: <, > <=, >=, =, <>

Comparison operators (eg. = or <=) return 1.0 when the condition is true and 0.0 otherwise. For instance, (a=b)+3 will evaluate to 4 when attribute ‘a’ is equal to attribute ‘b’, and to 3 when ‘a’ is not. Unlike MySQL, the equality comparisons (ie. = and <> operators) introduce a small equality threshold (1e-6 by default). If the difference between compared values is within the threshold, they will be considered equal.

• Boolean operators: AND, OR, NOT

Boolean operators (AND, OR, NOT) behave as usual. They are left-associative and have the least priority compared to other operators. NOT has more priority than AND and OR but nevertheless less than any other operator. AND and OR have the same priority so brackets use is recommended to avoid confusion in complex expressions.

• Bitwise operators: &, |

These operators perform bitwise AND and OR respectively. The operands must be of an integer types.

#### Numeric functions¶

• ABS()

Returns the absolute value of the argument.

• BITDOT()

BITDOT(mask, w0, w1, …) returns the sum of products of an each bit of a mask multiplied with its weight. bit0*w0 + bit1*w1 + ...

• BM25F()

BM25F(k1,b, {field=weight, …}) returns precise BM25F(). Requires expr ranker. k and b parameters must be float.

• CEIL()

Returns the smallest integer value greater or equal to the argument.

• CONTAINS()

CONTAINS(polygon, x, y) checks whether the (x,y) point is within the given polygon, and returns 1 if true, or 0 if false. The polygon has to be specified using either the POLY2D() function or the GEOPOLY2D() function. The former function is intended for “small” polygons, meaning less than 500 km (300 miles) a side, and it doesn’t take into account the Earth’s curvature for speed. For larger distances, you should use GEOPOLY2D, which tessellates the given polygon in smaller parts, accounting for the Earth’s curvature.

• COS()

Returns the cosine of the argument.

• DOUBLE() Forcibly promotes given argument to floating point type. Intended to help enforce evaluation of numeric JSON fields.
• EXP()

Returns the exponent of the argument (e=2.718… to the power of the argument).

• FIBONACCI()

Returns the N-th Fibonacci number, where N is the integer argument. That is, arguments of 0 and up will generate the values 0, 1, 1, 2, 3, 5, 8, 13 and so on. Note that the computations are done using 32-bit integer math and thus numbers 48th and up will be returned modulo 2^32.

• FLOOR()

Returns the largest integer value lesser or equal to the argument.

• GEOPOLY2D()

GEOPOLY2D(lat1,lon1,lat2,lon2,lat3,lon3…) produces a polygon to be used with the CONTAINS() function. This function takes into account the Earth’s curvature by tessellating the polygon into smaller ones, and should be used for larger areas. For small areas POLY2D() function can be used instead. The function expects coordinates to be pairs of latitude/longitude coordinates in degrees, if radians are used it will give same result as POLY2D().

• IDIV()

Returns the result of an integer division of the first argument by the second argument. Both arguments must be of an integer type.

• LN()

Returns the natural logarithm of the argument (with the base of e=2.718…).

• LOG10()

Returns the common logarithm of the argument (with the base of 10).

• LOG2()

Returns the binary logarithm of the argument (with the base of 2).

• MAX()

Returns the bigger of two arguments.

• MIN()

Returns the smaller of two arguments.

• POLY2D()

POLY2D(x1,y1,x2,y2,x3,y3…) produces a polygon to be used with the CONTAINS() function. This polygon assumes a flat Earth, so it should not be too large; for large areas the GEOPOLY2D() function which takes Earth’s curvature in consideration should be used.

• POW()

Returns the first argument raised to the power of the second argument.

• SIN()

Returns the sine of the argument.

• SQRT()

Returns the square root of the argument.

• UINT()

Forcibly reinterprets given argument to 64-bit unsigned type.

#### Date and time functions¶

• DAY()

Returns the integer day of month (in 1..31 range) from a timestamp argument, according to the current timezone.

• MONTH()

Returns the integer month (in 1..12 range) from a timestamp argument, according to the current timezone.

• NOW()

Returns the current timestamp as an INTEGER.

• YEAR()

Returns the integer year (in 1969..2038 range) from a timestamp argument, according to the current timezone.

• YEARMONTH()

Returns the integer year and month code (in 196912..203801 range) from a timestamp argument, according to the current timezone.

• YEARMONTHDAY()

Returns the integer year, month, and date code (in 19691231..20380119 range) from a timestamp argument, according to the current timezone.

• SECOND()

Returns the integer second (in 0..59 range) from a timestamp argument, according to the current timezone.

• MINUTE()

Returns the integer minute (in 0..59 range) from a timestamp argument, according to the current timezone.

• HOUR()

Returns the integer hour (in 0..23 range) from a timestamp argument, according to the current timezone.

#### Type conversion functions¶

• BIGINT()

Forcibly promotes the integer argument to 64-bit type, and does nothing on floating point argument. It’s intended to help enforce evaluation of certain expressions (such as a*b) in 64-bit mode even though all the arguments are 32-bit.

• INTEGER()

Forcibly promotes given argument to 64-bit signed type. Intended to help enforce evaluation of numeric JSON fields.

• SINT()

Forcibly reinterprets its 32-bit unsigned integer argument as signed, and also expands it to 64-bit type (because 32-bit type is unsigned). It’s easily illustrated by the following example: 1-2 normally evaluates to 4294967295, but SINT(1-2) evaluates to -1.

• TO_STRING()

Forcibly promotes the argument to string type.

#### Comparison functions¶

• IF()

IF() behavior is slightly different than its MySQL counterpart. It takes 3 arguments, checks whether the 1st argument is equal to 0.0, returns the 2nd argument if it is not zero or the 3rd one when it is. Note that unlike comparison operators, IF() does not use a threshold! Therefore, it’s safe to use comparison results as its 1st argument, but arithmetic operators might produce unexpected results. For instance, the following two calls will produce different results even though they are logically equivalent:

IF ( sqrt(3)*sqrt(3)-3<>0, a, b )
IF ( sqrt(3)*sqrt(3)-3, a, b )


In the first case, the comparison operator <> will return 0.0 (false) because of a threshold, and IF() will always return ‘**’ as a result. In the second one, the same sqrt(3)*sqrt(3)-3 expression will be compared with zero without threshold by the IF() function itself. But its value will be slightly different from zero because of limited floating point calculations precision. Because of that, the comparison with 0.0 done by IF() will not pass, and the second variant will return ‘a’ as a result.

• IN()

IN(expr,val1,val2,…) takes 2 or more arguments, and returns 1 if 1st argument (expr) is equal to any of the other arguments (val1..valN), or 0 otherwise. Currently, all the checked values (but not the expression itself!) are required to be constant. (Its technically possible to implement arbitrary expressions too, and that might be implemented in the future.) Constants are pre-sorted and then binary search is used, so IN() even against a big arbitrary list of constants will be very quick. First argument can also be a MVA attribute. In that case, IN() will return 1 if any of the MVA values is equal to any of the other arguments. IN() also supports IN(expr,@uservar) syntax to check whether the value belongs to the list in the given global user variable. First argument can be JSON attribute.

• INTERVAL()

INTERVAL(expr,point1,point2,point3,…), takes 2 or more arguments, and returns the index of the argument that is less than the first argument: it returns 0 if expr<point1, 1 if point1<=expr<point2, and so on. It is required that point1<point2<…<pointN for this function to work correctly.

#### Miscellaneous functions¶

• ALL()

ALL(cond FOR var IN json.array) applies to JSON arrays and returns 1 if condition is true for all elements in array and 0 otherwise. ‘cond’ is a general expression which additionally can use ‘var’ as current value of an array element within itself.

SELECT ALL(x>3 AND x<7 FOR x IN j.intarray) FROM test;


ALL(mva) is a special constructor for multi value attributes. When used in conjunction with comparison operators it returns 1 if all values compared are found among the MVA values.

SELECT * FROM test WHERE ALL(mymva)>10;


ALL(string list) is a special operation for filtering string tags.

SELECT * FROM test WHERE tags ALL('foo', 'bar', 'fake');
SELECT * FROM test WHERE tags NOT ALL('true', 'text', 'tag');


Here assumed that index ‘test’ has string attribute ‘tags’ with set of words (tags), separated by whitespace. If all of the words enumerated as arguments of ALL()’ present in the attribute, filter matches. Optional ‘NOT’ inverses the logic. For example, attr containing ‘buy iphone cheap’ will be matched by ALL('cheap', 'iphone'), but will not match ALL('iphone', '5s').

This filter internally uses doc-by-doc matching, so in case of full scan query it might be very slow. It is intended originally for attributes which are not indexed, like calculated expressions or tags in pq indexes.

if you like such filtering and want to use it in production, consider the solution to put the ‘tags’ attribute as full-text field, and then use FT operator ‘match()’ which will invoke full-text indexed search.

• ANY()

ANY(cond FOR var IN json.array) works similar to ALL() except for it returns 1 if condition is true for any element in array.

ANY(mva) is a special constructor for multi value attributes. When used in conjunction with comparison operators it returns 1 if any of the values compared are found among the MVA values. ANY is used by default if no constructor is used, however a warning will be raised about missing constructor.

ANY(string list) is a special operation for filtering string tags. Works similar to ALL(), except if condition is true for the case when any tag of tested expression match.

SELECT * FROM test WHERE tags NOT ANY('true', 'text', 'tag');
SELECT TO_STRING(id*321) secret FROM test WHERE secret ANY('1000','3210');

• ATAN2()

Returns the arctangent function of two arguments, expressed in radians.

• CONCAT()

Concatenates two or more strings into one. Non-string arguments must be explicitly converted to string using TO_STRING() function

CONCAT(TO_STRING(float_attr), ',', TO_STRING(int_attr), ',', title)

• CRC32()

Returns the CRC32 value of a string argument.

• GEODIST()

GEODIST(lat1, lon1, lat2, lon2, […]) function computes geosphere distance between two given points specified by their coordinates. Note that by default both latitudes and longitudes must be in radians and the result will be in meters. You can use arbitrary expression as any of the four coordinates. An optimized path will be selected when one pair of the arguments refers directly to a pair attributes and the other one is constant.

GEODIST() also takes an optional 5th argument that lets you easily convert between input and output units, and pick the specific geodistance formula to use. The complete syntax and a few examples are as follows:

GEODIST(lat1, lon1, lat2, lon2, { option=value, ... })

GEODIST(40.7643929, -73.9997683, 40.7642578, -73.9994565, {in=degrees, out=feet})

GEODIST(51.50, -0.12, 29.98, 31.13, {in=deg, out=mi}}


The known options and their values are:

• in = {deg | degrees | rad | radians}, specifies the input units;
• out = {m | meters | km | kilometers | ft | feet | mi | miles}, specifies the output units;
• method = {adaptive | haversine}, specifies the geodistance calculation method.

The default method is “adaptive”. It is well optimized implementation that is both more precise and much faster at all times than “haversine”.

• GREATEST()

GREATEST(attr_json.some_array) function takes JSON array as the argument, and returns the greatest value in that array. Also works for MVA.

• INDEXOF()

INDEXOF(cond FOR var IN json.array) function iterates through all elements in array and returns index of first element for which ‘cond’ is true and -1 if ‘cond’ is false for every element in array.

SELECT INDEXOF(name='John' FOR name IN j.peoples) FROM test;

• LEAST()

LEAST(attr_json.some_array) function takes JSON array as the argument, and returns the least value in that array. Also works for MVA.

• LENGTH()

LENGTH(attr_mva) function returns amount of elements in MVA set. It works with both 32-bit and 64-bit MVA attributes. LENGTH(attr_json) returns length of a field in JSON. Return value depends on type of a field. For example LENGTH(json_attr.some_int) always returns 1 and LENGTH(json_attr.some_array) returns number of elements in array.

• MIN_TOP_SORTVAL()

Returns sort key value of the worst found element in the current top-N matches if sort key is float and 0 otherwise.

• MIN_TOP_WEIGHT() Returns weight of the worst found element in the current top-N matches.
• PACKEDFACTORS()

PACKEDFACTORS() can be used in queries, either to just see all the weighting factors calculated when doing the matching, or to provide a binary attribute that can be used to write a custom ranking UDF. This function works only if expression ranker is specified and the query is not a full scan, otherwise it will return an error. PACKEDFACTORS() can take an optional argument that disables ATC ranking factor calculation:

PACKEDFACTORS({no_atc=1})


Calculating ATC slows down query processing considerably, so this option can be useful if you need to see the ranking factors, but do not need ATC. PACKEDFACTORS() can also be told to format its output as JSON:

PACKEDFACTORS({json=1})


The respective outputs in either key-value pair or JSON format would look as follows below. (Note that the examples below are wrapped for readability; actual returned values would be single-line.)

mysql> SELECT id, PACKEDFACTORS() FROM test1
-> WHERE MATCH('test one') OPTION ranker=expr('1') \G
*************************** 1\. row ***************************
id: 1
field1=(lcs=1, hit_count=2, word_count=2, tf_idf=0.152356,
min_idf=-0.062982, max_idf=0.215338, sum_idf=0.152356, min_hit_pos=4,
min_best_span_pos=4, exact_hit=0, max_window_hits=1, min_gaps=2,
exact_order=1, lccs=1, wlccs=0.215338, atc=-0.003974),
word0=(tf=1, idf=-0.062982),
word1=(tf=1, idf=0.215338)
1 row in set (0.00 sec)

mysql> SELECT id, PACKEDFACTORS({json=1}) FROM test1
-> WHERE MATCH('test one') OPTION ranker=expr('1') \G
*************************** 1\. row ***************************
id: 1
packedfactors({json=1}):
{

"bm25": 569,
"bm25a": 0.617197,
"doc_word_count": 2,
"fields": [
{
"lcs": 1,
"hit_count": 2,
"word_count": 2,
"tf_idf": 0.152356,
"min_idf": -0.062982,
"max_idf": 0.215338,
"sum_idf": 0.152356,
"min_hit_pos": 4,
"min_best_span_pos": 4,
"exact_hit": 0,
"max_window_hits": 1,
"min_gaps": 2,
"exact_order": 1,
"lccs": 1,
"wlccs": 0.215338,
"atc": -0.003974
}
],
"words": [
{
"tf": 1,
"idf": -0.062982
},
{
"tf": 1,
"idf": 0.215338
}
]

}
1 row in set (0.01 sec)


This function can be used to implement custom ranking functions in UDFs, as in

SELECT *, CUSTOM_RANK(PACKEDFACTORS()) AS r
FROM my_index
WHERE match('hello')
ORDER BY r DESC
OPTION ranker=expr('1');


Where CUSTOM_RANK() is a function implemented in an UDF. It should declare a SPH_UDF_FACTORS structure (defined in sphinxudf.h), initialize this structure, unpack the factors into it before usage, and deinitialize it afterwards, as follows:

SPH_UDF_FACTORS factors;
sphinx_factors_init(&factors);
sphinx_factors_unpack((DWORD*)args->arg_values[0], &factors);
// ... can use the contents of factors variable here ...
sphinx_factors_deinit(&factors);


PACKEDFACTORS() data is available at all query stages, not just when doing the initial matching and ranking pass. That enables another particularly interesting application of PACKEDFACTORS(), namely re-ranking.

In the example just above, we used an expression-based ranker with a dummy expression, and sorted the result set by the value computed by our UDF. In other words, we used the UDF to rank all our results. Assume now, for the sake of an example, that our UDF is extremely expensive to compute and has a throughput of just 10,000 calls per second. Assume that our query matches 1,000,000 documents. To maintain reasonable performance, we would then want to use a (much) simpler expression to do most of our ranking, and then apply the expensive UDF to only a few top results, say, top-100 results. Or, in other words, build top-100 results using a simpler ranking function and then re-rank those with a complex one. We can do that just as well with subselects:

SELECT * FROM (
SELECT *, CUSTOM_RANK(PACKEDFACTORS()) AS r
FROM my_index WHERE match('hello')
OPTION ranker=expr('sum(lcs)*1000+bm25')
ORDER BY WEIGHT() DESC
LIMIT 100
) ORDER BY r DESC LIMIT 10


In this example, expression-based ranker will be called for every matched document to compute WEIGHT(). So it will get called 1,000,000 times. But the UDF computation can be postponed until the outer sort. And it also will be done for just the top-100 matches by WEIGHT(), according to the inner limit. So the UDF will only get called 100 times. And then the final top-10 matches by UDF value will be selected and returned to the application.

For reference, in the distributed case PACKEDFACTORS() data gets sent from the agents to master in a binary format, too. This makes it technically feasible to implement additional re-ranking pass (or passes) on the master node, if needed.

If used with SphinxQL but not called from any UDFs, the result of PACKEDFACTORS() is simply formatted as plain text, which can be used to manually assess the ranking factors. Note that this feature is not currently supported by the Manticore API.

• REMAP()

REMAP(condition, expression, (cond1, cond2, …), (expr1, expr2, …)) function allows you to make some exceptions of an expression values depending on condition values. Condition expression should always result integer, expression can result in integer or float.

SELECT REMAP(userid, karmapoints, (1, 67), (999, 0)) FROM users;
SELECT REMAP(id%10, salary, (0), (0.0)) FROM employes;

• RAND()

RAND(seed) function returns a random float between 0..1. Optional, an integer seed value can be specified.

• REGEX()

REGEX(attr,expr) function returns 1 if regular expression matched to string of attribute and 0 otherwise. It works with both string and JSON attributes.

We use the RE2 engine to implement regexps. So when building from the source, the library must be installed in the system and Manticore must be configured built with a –with-re2 switch. Binary packages should come with RE2 builtin.

SELECT REGEX(content, 'box?') FROM test;
SELECT REGEX(j.color, 'red | pink') FROM test;

• SUBSTRING_INDEX()

SUBSTRING_INDEX(string, delimiter, number) returns a substring of a string before a specified number of delimiter occurs

• string - The original string. Can be a constant string or a string from a string/json attribute.
• delimiter - The delimiter to search for
• number - The number of times to search for the delimiter. Can be both a positive or negative number.If it is a positive number, this function will return all to the left of the delimiter. If it is a negative number, this function will return all to the right of the delimiter.
SELECT SUBSTRING_INDEX('www.w3schools.com', '.', 2) FROM test;
SELECT SUBSTRING_INDEX(j.coord, ' ', 1) FROM test;

• WEIGHT() Returns fulltext match score.

### Sorting modes¶

There are the following result sorting modes available:

• SPH_SORT_RELEVANCE mode, that sorts by relevance in descending order (best matches first);
• SPH_SORT_ATTR_DESC mode, that sorts by an attribute in descending order (bigger attribute values first);
• SPH_SORT_ATTR_ASC mode, that sorts by an attribute in ascending order (smaller attribute values first);
• SPH_SORT_TIME_SEGMENTS mode, that sorts by time segments (last hour/day/week/month) in descending order, and then by relevance in descending order;
• SPH_SORT_EXTENDED mode, that sorts by SQL-like combination of columns in ASC/DESC order;
• SPH_SORT_EXPR mode, that sorts by an arithmetic expression.

SPH_SORT_RELEVANCE ignores any additional parameters and always sorts matches by relevance rank. All other modes require an additional sorting clause, with the syntax depending on specific mode. SPH_SORT_ATTR_ASC, SPH_SORT_ATTR_DESC and SPH_SORT_TIME_SEGMENTS modes require simply an attribute name. SPH_SORT_RELEVANCE is equivalent to sorting by “@weight DESC, @id ASC” in extended sorting mode, SPH_SORT_ATTR_ASC is equivalent to “attribute ASC, @weight DESC, @id ASC”, and SPH_SORT_ATTR_DESC to “attribute DESC, @weight DESC, @id ASC” respectively.

#### SPH_SORT_TIME_SEGMENTS mode¶

In SPH_SORT_TIME_SEGMENTS mode, attribute values are split into so-called time segments, and then sorted by time segment first, and by relevance second.

The segments are calculated according to the current timestamp at the time when the search is performed, so the results would change over time. The segments are as follows:

• last hour,
• last day,
• last week,
• last month,
• last 3 months,
• everything else.

These segments are hardcoded, but it is trivial to change them if necessary.

This mode was added to support searching through blogs, news headlines, etc. When using time segments, recent records would be ranked higher because of segment, but within the same segment, more relevant records would be ranked higher - unlike sorting by just the timestamp attribute, which would not take relevance into account at all.

#### SPH_SORT_EXTENDED mode¶

In SPH_SORT_EXTENDED mode, you can specify an SQL-like sort expression with up to 5 attributes (including internal attributes), eg:

@relevance DESC, price ASC, @id DESC


Both internal attributes (that are computed by the engine on the fly) and user attributes that were configured for this index are allowed. Internal attribute names must start with magic @-symbol; user attribute names can be used as is. In the example above, @relevance and @id are internal attributes and price is user-specified.

Known internal attributes are:

• @id (match ID)
• @weight (match weight)
• @rank (match weight)
• @relevance (match weight)
• @random (return results in random order)

@rank and @relevance are just additional aliases to @weight.

#### SPH_SORT_EXPR mode¶

Expression sorting mode lets you sort the matches by an arbitrary arithmetic expression, involving attribute values, internal attributes (@id and @weight), arithmetic operations, and a number of built-in functions. Here’s an example:

$cl->SetSortMode ( SPH_SORT_EXPR, "@weight + ( user_karma + ln(pageviews) )*0.1" );  The operators and functions supported in the expressions are discussed in Expressions, functions, and operators. ### Grouping (clustering) search results¶ Sometimes it could be useful to group (or in other terms, cluster) search results and/or count per-group match counts - for instance, to draw a nice graph of how much matching blog posts were there per each month; or to group Web search results by site; or to group matching forum posts by author; etc. In theory, this could be performed by doing only the full-text search in Manticore and then using found IDs to group on SQL server side. However, in practice doing this with a big result set (10K-10M matches) would typically kill performance. To avoid that, Manticore offers so-called grouping mode. It is enabled with SetGroupBy() API call. When grouping, all matches are assigned to different groups based on group-by value. This value is computed from specified attribute using one of the following built-in functions: • SPH_GROUPBY_DAY, extracts year, month and day in YYYYMMDD format from timestamp; • SPH_GROUPBY_WEEK, extracts year and first day of the week number (counting from year start) in YYYYNNN format from timestamp; • SPH_GROUPBY_MONTH, extracts month in YYYYMM format from timestamp; • SPH_GROUPBY_YEAR, extracts year in YYYY format from timestamp; • SPH_GROUPBY_ATTR, uses attribute value itself for grouping. The final search result set then contains one best match per group. Grouping function value and per-group match count are returned along as “virtual” attributes named @group and @count respectively. The result set is sorted by group-by sorting clause, with the syntax similar to SPH_SORT_EXTENDED sorting clause syntax. In addition to @id and @weight, group-by sorting clause may also include: • @group (groupby function value), • @count (amount of matches in group). The default mode is to sort by groupby value in descending order, ie. by @group desc. On completion, total_found result parameter would contain total amount of matching groups over he whole index. WARNING: grouping is done in fixed memory and thus its results are only approximate; so there might be more groups reported in total_found than actually present. @count might also be underestimated. To reduce inaccuracy, one should raise max_matches. If max_matches allows to store all found groups, results will be 100% correct. For example, if sorting by relevance and grouping by "published" attribute with SPH_GROUPBY_DAY function, then the result set will contain • one most relevant match per each day when there were any matches published, • with day number and per-day match count attached, • sorted by day number in descending order (ie. recent days first). Aggregate functions (AVG(), MIN(), MAX(), SUM()) are supported through SetSelect() API call when using GROUP BY. ### Distributed searching¶ To scale well, Manticore has distributed searching capabilities. Distributed searching is useful to improve query latency (ie. search time) and throughput (ie. max queries/sec) in multi-server, multi-CPU or multi-core environments. This is essential for applications which need to search through huge amounts data (ie. billions of records and terabytes of text). The key idea is to horizontally partition (HP) searched data across search nodes and then process it in parallel. Partitioning is done manually. You should • setup several instances of Manticore programs (indexer and searchd) on different servers; • make the instances index (and search) different parts of data; • configure a special distributed index on some of the searchd instances; • and query this index. This index only contains references to other local and remote indexes - so it could not be directly reindexed, and you should reindex those indexes which it references instead. When searchd receives a query against distributed index, it does the following: 1. connects to configured remote agents; 2. issues the query; 3. sequentially searches configured local indexes (while the remote agents are searching); 4. retrieves remote agents’ search results; 5. merges all the results together, removing the duplicates; 6. sends the merged results to client. From the application’s point of view, there are no differences between searching through a regular index, or a distributed index at all. That is, distributed indexes are fully transparent to the application, and actually there’s no way to tell whether the index you queried was distributed or local. Any searchd instance could serve both as a master (which aggregates the results) and a slave (which only does local searching) at the same time. This has a number of uses: 1. every machine in a cluster could serve as a master which searches the whole cluster, and search requests could be balanced between masters to achieve a kind of HA (high availability) in case any of the nodes fails; 2. if running within a single multi-CPU or multi-core machine, there would be only 1 searchd instance querying itself as an agent and thus utilizing all CPUs/core. It is scheduled to implement better HA support which would allow to specify which agents mirror each other, do health checks, keep track of alive agents, load-balance requests, etc. ### Query log formats¶ Two query log formats are supported. Plain text format is still the default one. However, while it might be more convenient for manual monitoring and review, but hard to replay for benchmarks, it only logs search queries but not the other types of requests, does not always contain the complete search query data, etc. The default text format is also harder (and sometimes impossible) to replay for benchmarking purposes. The sphinxql format alleviates that. It aims to be complete and automatable, even though at the cost of brevity and readability. #### Plain log format¶ By default, searchd logs all successfully executed search queries into a query log file. Here’s an example: [Fri Jun 29 21:17:58 2007] 0.004 sec 0.004 sec [all/0/rel 35254 (0,20)] [lj] test [Fri Jun 29 21:20:34 2007] 0.024 sec 0.024 sec [all/0/rel 19886 (0,20) @channel_id] [lj] test  This log format is as follows: [query-date] real-time wall-time [match-mode/filters-count/sort-mode total-matches (offset,limit) @groupby-attr] [index-name] query  • real-time is a time measured just from start to finish of the query • wall-time like real-time but not including waiting for agents and merging result sets time Match mode can take one of the following values: • “all” for SPH_MATCH_ALL mode; • “any” for SPH_MATCH_ANY mode; • “phr” for SPH_MATCH_PHRASE mode; • “bool” for SPH_MATCH_BOOLEAN mode; • “ext” for SPH_MATCH_EXTENDED mode; • “ext2” for SPH_MATCH_EXTENDED2 mode; • “scan” if the full scan mode was used, either by being specified with SPH_MATCH_FULLSCAN, or if the query was empty (as documented under Matching Modes) Sort mode can take one of the following values: • “rel” for SPH_SORT_RELEVANCE mode; • “attr-” for SPH_SORT_ATTR_DESC mode; • “attr+” for SPH_SORT_ATTR_ASC mode; • “tsegs” for SPH_SORT_TIME_SEGMENTS mode; • “ext” for SPH_SORT_EXTENDED mode. Additionally, if searchd was started with --iostats, there will be a block of data after where the index(es) searched are listed. A query log entry might take the form of: [Fri Jun 29 21:17:58 2007] 0.004 sec [all/0/rel 35254 (0,20)] [lj] [ios=6 kb=111.1 ms=0.5] test  This additional block is information regarding I/O operations in performing the search: the number of file I/O operations carried out, the amount of data in kilobytes read from the index files and time spent on I/O operations (although there is a background processing component, the bulk of this time is the I/O operation time). #### SphinxQL log format¶ This new log format introduced with the goals begin logging everything and then some, and in a format easy to automate (for instance, automatically replay). SphinxQL log format can either be enabled via the query_log_format directive in the configuration file, or switched back and forth on the fly with the SET GLOBAL query_log_format=… statement via SphinxQL. In the new format, the example from the previous section would look as follows. (Wrapped below for readability, but with just one query per line in the actual log.) /* Fri Jun 29 21:17:58.609 2007 2011 conn 2 real 0.004 wall 0.004 found 35254 */ SELECT * FROM lj WHERE MATCH('test') OPTION ranker=proximity; /* Fri Jun 29 21:20:34 2007.555 conn 3 real 0.024 wall 0.024 found 19886 */ SELECT * FROM lj WHERE MATCH('test') GROUP BY channel_id OPTION ranker=proximity;  Note that all requests would be logged in this format, including those sent via SphinxAPI and SphinxSE, not just those sent via SphinxQL. Also note, that this kind of logging works only with plain log files and will not work if you use ‘syslog’ service for logging. The features of SphinxQL log format compared to the default text one are as follows. • All request types should be logged. (This is still work in progress.) • Full statement data will be logged where possible. • Errors and warnings are logged. • The log should be automatically replayable via SphinxQL. • Additional performance counters (currently, per-agent distributed query times) are logged. Use sphinxql:compact_in to shorten your IN() clauses in log if you have too much values in it. Every request (including both SphinxAPI and SphinxQL) request must result in exactly one log line. All request types, including INSERT, CALL SNIPPETS, etc will eventually get logged, though as of time of this writing, that is a work in progress). Every log line must be a valid SphinxQL statement that reconstructs the full request, except if the logged request is too big and needs shortening for performance reasons. Additional messages, counters, etc can be logged in the comments section after the request. ### MySQL protocol support and SphinxQL¶ Manticore searchd daemon supports MySQL binary network protocol and can be accessed with regular MySQL API. For instance, ‘mysql’ CLI client program works well. Here’s an example of querying Manticore using MySQL client: $ mysql -P 9306
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 0.9.9-dev (r1734)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> SELECT * FROM test1 WHERE MATCH('test')
-> ORDER BY group_id ASC OPTION ranker=bm25;
+------+--------+----------+------------+
| id   | weight | group_id | date_added |
+------+--------+----------+------------+
|    4 |   1442 |        2 | 1231721236 |
|    2 |   2421 |      123 | 1231721236 |
|    1 |   2421 |      456 | 1231721236 |
+------+--------+----------+------------+
3 rows in set (0.00 sec)


Note that mysqld was not even running on the test machine. Everything was handled by searchd itself.

The new access method is supported in addition to native APIs which all still work perfectly well. In fact, both access methods can be used at the same time. Also, native API is still the default access method. MySQL protocol support needs to be additionally configured. This is a matter of 1-line config change, adding a new listener with mysql41 specified as a protocol:

listen = localhost:9306:mysql41


Just supporting the protocol and not the SQL syntax would be useless so Manticore now also supports a subset of SQL that we dubbed SphinxQL. It supports the standard querying all the index types with SELECT, modifying RT indexes with INSERT, REPLACE, and DELETE, and much more. Full SphinxQL reference is available in SphinxQL reference.

### Multi-queries¶

Multi-queries, or query batches, let you send multiple queries to Manticore in one go (more formally, one network request).

Two API methods that implement multi-query mechanism are AddQuery() and RunQueries(). You can also run multiple queries with SphinxQL, see Multi-statement queries. (In fact, regular Query() call is internally implemented as a single AddQuery() call immediately followed by RunQueries() call.) AddQuery() captures the current state of all the query settings set by previous API calls, and memorizes the query. RunQueries() actually sends all the memorized queries, and returns multiple result sets. There are no restrictions on the queries at all, except just a sanity check on a number of queries in a single batch (see max_batch_queries).

Why use multi-queries? Generally, it all boils down to performance. First, by sending requests to searchd in a batch instead of one by one, you always save a bit by doing less network roundtrips. Second, and somewhat more important, sending queries in a batch enables searchd to perform certain internal optimizations. As new types of optimizations are being added over time, it generally makes sense to pack all the queries into batches where possible, so that simply upgrading Manticore to a new version would automatically enable new optimizations. In the case when there aren’t any possible batch optimizations to apply, queries will be processed one by one internally.

Why (or rather when) not use multi-queries? Multi-queries requires all the queries in a batch to be independent, and sometimes they aren’t. That is, sometimes query B is based on query A results, and so can only be set up after executing query A. For instance, you might want to display results from a secondary index if and only if there were no results found in a primary index. Or maybe just specify offset into 2nd result set based on the amount of matches in the 1st result set. In that case, you will have to use separate queries (or separate batches).

There are two major optimizations to be aware of: common query optimization and common subtree optimization.

Common query optimization means that searchd will identify all those queries in a batch where only the sorting and group-by settings differ, and only perform searching once. For instance, if a batch consists of 3 queries, all of them are for “ipod nano”, but 1st query requests top-10 results sorted by price, 2nd query groups by vendor ID and requests top-5 vendors sorted by rating, and 3rd query requests max price, full-text search for “ipod nano” will only be performed once, and its results will be reused to build 3 different result sets.

So-called faceted searching is a particularly important case that benefits from this optimization. Indeed, faceted searching can be implemented by running a number of queries, one to retrieve search results themselves, and a few other ones with same full-text query but different group-by settings to retrieve all the required groups of results (top-3 authors, top-5 vendors, etc). And as long as full-text query and filtering settings stay the same, common query optimization will trigger, and greatly improve performance.

Common subtree optimization is even more interesting. It lets searchd exploit similarities between batched full-text queries. It identifies common full-text query parts (subtrees) in all queries, and caches them between queries. For instance, look at the following query batch:

donald trump president
donald trump barack obama john mccain
donald trump speech


There’s a common two-word part (“donald trump”) that can be computed only once, then cached and shared across the queries. And common subtree optimization does just that. Per-query cache size is strictly controlled by subtree_docs_cache and subtree_hits_cache directives (so that caching all sixteen gazillions of documents that match “i am” does not exhaust the RAM and instantly kill your server).

Here’s a code sample (in PHP) that fire the same query in 3 different sorting modes:

require ( "sphinxapi.php" );
$cl = new ManticoreClient ();$cl->SetMatchMode ( SPH_MATCH_EXTENDED );

$cl->SetSortMode ( SPH_SORT_RELEVANCE );$cl->AddQuery ( "the", "lj" );
$cl->SetSortMode ( SPH_SORT_EXTENDED, "published desc" );$cl->AddQuery ( "the", "lj" );
$cl->SetSortMode ( SPH_SORT_EXTENDED, "published asc" );$cl->AddQuery ( "the", "lj" );
$res =$cl->RunQueries();


How to tell whether the queries in the batch were actually optimized? If they were, respective query log will have a “multiplier” field that specifies how many queries were processed together:

[Sun Jul 12 15:18:17.000 2009] 0.040 sec x3 [ext/0/rel 747541 (0,20)] [lj] the
[Sun Jul 12 15:18:17.000 2009] 0.040 sec x3 [ext/0/ext 747541 (0,20)] [lj] the
[Sun Jul 12 15:18:17.000 2009] 0.040 sec x3 [ext/0/ext 747541 (0,20)] [lj] the


Note the “x3” field. It means that this query was optimized and processed in a sub-batch of 3 queries. For reference, this is how the regular log would look like if the queries were not batched:

[Sun Jul 12 15:18:17.062 2009] 0.059 sec [ext/0/rel 747541 (0,20)] [lj] the
[Sun Jul 12 15:18:17.156 2009] 0.091 sec [ext/0/ext 747541 (0,20)] [lj] the
[Sun Jul 12 15:18:17.250 2009] 0.092 sec [ext/0/ext 747541 (0,20)] [lj] the


Note how per-query time in multi-query case was improved by a factor of 1.5x to 2.3x, depending on a particular sorting mode. In fact, for both common query and common subtree optimizations, there were reports of 3x and even more improvements, and that’s from production instances, not just synthetic tests.

### Geo Searching¶

One of the greatest features of Manticore Search is the ability to combine full-text searching with geo location. For example a retailer can offer a search where user looks for a product and the result set can tell which is the closest shop that has the product in stock so user can go in store and pick it up. A travel site can provide results based on a search limited to a certain area and results to be sorted by the distance from a point (‘search museums near a hotel’ for example).

To perform geo searching, a document needs to contain pairs of latitude/longitude coordinates. The coordinates can be stored as float attributes. If the document has multiple locations, they can be embedded in a JSON object as float pairs.

index myrt {
...
rt_attr_float = lat
rt_attr_float = lon
...
}


The coordinates can be stored as degrees or radians.

#### Performing distance calculation¶

To find out the distance between two points the GEODIST() function can be used. GEODIST requires two pairs of coordinates as first four parameters.

A 5th parameter in a simplified JSON format can configure certain aspects of the function. By default, GEODIST expects coordinates to be in radians, but in=degrees can be added to allow using degrees at input. The coordinates for which we perform the geo distance must have same time (degrees or radians) as the ones stored in the index, otherwise results will be misleading.

The calculated distance is by default in meters, but with out option it can be transformed to kilometers, feets or miles. Lastly, by default a calculation method called adaptive is used. An alternative method based on haversine algorithm is available, however this one is slower and less precise.

The result of the function - the distance - can be used in ORDER BY clause to sort the results

SELECT *,GEODIST(40.7643929, -73.9997683, lat,lon, {in=degrees, out=miles}) AS distance FROM myindex WHERE MATCH('...') ORDER BY distance ASC, WEIGHT() DESC;


Or to limit the results to a radial area around the point:

SELECT *,GEODIST(40.7643929, -73.9997683, lat,lon, {in=degrees, out=miles}) AS distance FROM myindex WHERE MATCH('...') AND distance <1000 ORDER BY  WEIGHT(),DISTANCE ASC;


#### Searching in polygons¶

Another geo search functionality is the ability to check if a location belongs to an area. A special function will construct a polygon object which is used in another function that test if a set of coordinates belongs to it or not.

For creating the polygon two functions are available:

• GEOPOLY2D() - creates a polygon that takes in account the Earth’s curvature
• POLY2D() - creates a simple polygon in plain space

POLY2D can be used for geo searches if the area has sides shorter than 500km (for 3-4 sides, for polygons with more sides lower values should be considered). For areas with longer sides usage of GEOPOLY2D is mandatory for keeping results accurate. GEOPOLY2D also expects coordinates as latitude/longitude pairs in degrees, using radians will provide results in plain space (like POLY2D).

CONTAINS() expects at input a polygon and a set of coordinates and output 1 if the point is inside the polygon or 0 otherwise.

SELECT *,CONTAINS(GEOPOLY2D(40.76439, -73.9997, 42.21211, -73.999,  42.21211, -76.123, 40.76439 , -76.123), 41.5445,-74.973) AS inside FROM myindex WHERE MATCH('...') AND inside=1;


### Collations¶

Collations essentially affect the string attribute comparisons. They specify both the character set encoding and the strategy that Manticore uses to compare strings when doing ORDER BY or GROUP BY with a string attribute involved.

String attributes are stored as is when indexing, and no character set or language information is attached to them. That’s okay as long as Manticore only needs to store and return the strings to the calling application verbatim. But when you ask Manticore to sort by a string value, that request immediately becomes quite ambiguous.

First, single-byte (ASCII, or ISO-8859-1, or Windows-1251) strings need to be processed differently that the UTF-8 ones that may encode every character with a variable number of bytes. So we need to know what is the character set type to interpret the raw bytes as meaningful characters properly.

Second, we additionally need to know the language-specific string sorting rules. For instance, when sorting according to US rules in en_US locale, the accented character ‘ï’ (small letter i with diaeresis) should be placed somewhere after ‘z’. However, when sorting with French rules and fr_FR locale in mind, it should be placed between ‘i’ and ‘j’. And some other set of rules might choose to ignore accents at all, allowing ‘ï’ and ‘i’ to be mixed arbitrarily.

Third, but not least, we might need case-sensitive sorting in some scenarios and case-insensitive sorting in some others.

Collations combine all of the above: the character set, the language rules, and the case sensitivity. Manticore currently provides the following four collations.

1. libc_ci
2. libc_cs
3. utf8_general_ci
4. binary

The first two collations rely on several standard C library (libc) calls and can thus support any locale that is installed on your system. They provide case-insensitive (_ci) and case-sensitive (_cs) comparisons respectively. By default they will use C locale, effectively resorting to bytewise comparisons. To change that, you need to specify a different available locale using collation_libc_locale directive. The list of locales available on your system can usually be obtained with the locale command:

$locale -a C en_AG en_AU.utf8 en_BW.utf8 en_CA.utf8 en_DK.utf8 en_GB.utf8 en_HK.utf8 en_IE.utf8 en_IN en_NG en_NZ.utf8 en_PH.utf8 en_SG.utf8 en_US.utf8 en_ZA.utf8 en_ZW.utf8 es_ES fr_FR POSIX ru_RU.utf8 ru_UA.utf8  The specific list of the system locales may vary. Consult your OS documentation to install additional needed locales. utf8_general_ci and binary locales are built-in into Manticore. The first one is a generic collation for UTF-8 data (without any so-called language tailoring); it should behave similar to utf8_general_ci collation in MySQL. The second one is a simple bytewise comparison. Collation can be overridden via SphinxQL on a per-session basis using SET collation_connection statement. All subsequent SphinxQL queries will use this collation. SphinxAPI and SphinxSE queries will use the server default collation, as specified in collation_server configuration directive. Manticore currently defaults to libc_ci collation. Collations should affect all string attribute comparisons, including those within ORDER BY and GROUP BY, so differently ordered or grouped results can be returned depending on the collation chosen. Note that collations don’t affect full-text searching, for that use charset_table. ### Query cache¶ Query cache stores a compressed result set in memory, and then reuses it for subsequent queries where possible. You can configure it using the following directives: • qcache_max_bytes, a limit on the RAM use for cached queries storage. Defaults to 16 MB. Setting qcache_max_bytes to 0 completely disables the query cache. • qcache_thresh_msec, the minimum wall query time to cache. Queries that completed faster than this will not be cached. Defaults to 3000 msec, or 3 seconds. • qcache_ttl_sec, cached entry TTL, or time to live. Queries will stay cached for this much. Defaults to 60 seconds, or 1 minute. These settings can be changed on the fly using the SET GLOBAL statement: mysql> SET GLOBAL qcache_max_bytes=128000000;  These changes are applied immediately, and the cached result sets that no longer satisfy the constraints are immediately discarded. When reducing the cache size on the fly, MRU (most recently used) result sets win. Query cache works as follows. When it’s enabled, every full-text search result gets completely stored in memory. That happens after full-text matching, filtering, and ranking, so basically we store total_found {docid,weight} pairs. Compressed matches can consume anywhere from 2 bytes to 12 bytes per match on average, mostly depending on the deltas between the subsequent docids. Once the query completes, we check the wall time and size thresholds, and either save that compressed result set for reuse, or discard it. Note how the query cache impact on RAM is thus not limited by qcache_max_bytes! If you run, say, 10 concurrent queries, each of them matching upto 1M matches (after filters), then the peak temporary RAM use will be in the 40 MB to 240 MB range, even if in the end the queries are quick enough and do not get cached. Queries can then use cache when the index, the full-text query (ie. MATCH() contents), and the ranker are all a match, and filters are compatible. Meaning: • The full-text part within MATCH() must be a bytewise match. Add a single extra space, and that is now a different query where the query cache is concerned. • The ranker (and its parameters if any, for user-defined rankers) must be a bytewise match. • The filters must be a superset of the original filters. That is, you can add extra filters and still hit the cache. (In this case, the extra filters will be applied to the cached result.) But if you remove one, that will be a new query again. Cache entries expire with TTL, and also get invalidated on index rotation, or on TRUNCATE, or on ATTACH. Note that at the moment entries are not invalidated on arbitrary RT index writes! So a cached query might be returning older results for the duration of its TTL. Current cache status can be inspected with in SHOW STATUS through the qcache_XXX variables: mysql> SHOW STATUS LIKE 'qcache%'; +-----------------------+----------+ | Counter | Value | +-----------------------+----------+ | qcache_max_bytes | 16777216 | | qcache_thresh_msec | 3000 | | qcache_ttl_sec | 60 | | qcache_cached_queries | 0 | | qcache_used_bytes | 0 | | qcache_hits | 0 | +-----------------------+----------+ 6 rows in set (0.00 sec)  ### MySQL storage engine (SphinxSE)¶ #### SphinxSE overview¶ SphinxSE is MySQL storage engine which can be compiled into MySQL server 5.x using its pluggable architecture. It is not available for MySQL 4.x series. It also requires MySQL 5.0.22 or higher in 5.0.x series, or MySQL 5.1.12 or higher in 5.1.x series. Despite the name, SphinxSE does not actually store any data itself. It is actually a built-in client which allows MySQL server to talk to searchd, run search queries, and obtain search results. All indexing and searching happen outside MySQL. Obvious SphinxSE applications include: • easier porting of MySQL FTS applications to Manticore; • allowing Manticore use with programming languages for which native APIs are not available yet; • optimizations when additional Manticore result set processing on MySQL side is required (eg. JOINs with original document tables, additional MySQL-side filtering, etc). ##### Installing SphinxSE¶ You will need to obtain a copy of MySQL sources, prepare those, and then recompile MySQL binary. MySQL sources (mysql-5.x.yy.tar.gz) could be obtained from http://dev.mysql.com Web site. For some MySQL versions, there are delta tarballs with already prepared source versions available from Manticore Web site. After unzipping those over original sources MySQL would be ready to be configured and built with Manticore support. If such tarball is not available, or does not work for you for any reason, you would have to prepare sources manually. You will need to GNU Autotools framework (autoconf, automake and libtool) installed to do that. ##### Compiling MySQL 5.0.x with SphinxSE¶ 1. copy sphinx.5.0.yy.diff patch file into MySQL sources directory and run $ patch -p1 < sphinx.5.0.yy.diff

If there’s no .diff file exactly for the specific version you need to
build, try applying .diff with closest version numbers. It is important that the patch should apply with no rejects.
1. in MySQL sources directory, run
$sh BUILD/autorun.sh  1. in MySQL sources directory, create sql/sphinx directory in and copy all files in mysqlse directory from Manticore sources there. Example: $ cp -R /root/builds/sphinx-0.9.7/mysqlse /root/builds/mysql-5.0.24/sql/sphinx

1. configure MySQL and enable Manticore engine:
$./configure --with-sphinx-storage-engine  1. build and install MySQL: $ make
$make install  ##### Compiling MySQL 5.1.x with SphinxSE¶ 1. in MySQL sources directory, create storage/sphinx directory in and copy all files in mysqlse directory from Manticore sources there. Example: $ cp -R /root/builds/sphinx-0.9.7/mysqlse /root/builds/mysql-5.1.14/storage/sphinx

1. in MySQL sources directory, run
$sh BUILD/autorun.sh  1. configure MySQL and enable Manticore engine: $ ./configure --with-plugins=sphinx

1. build and install MySQL:
$make$ make install

##### Checking SphinxSE installation¶

To check whether SphinxSE has been successfully compiled into MySQL, launch newly built servers, run mysql client and issue SHOW ENGINES query. You should see a list of all available engines. Manticore should be present and “Support” column should contain “YES”:

mysql> show engines;
+------------+----------+-------------------------------------------------------------+
| Engine     | Support  | Comment                                                     |
+------------+----------+-------------------------------------------------------------+
| MyISAM     | DEFAULT  | Default engine as of MySQL 3.23 with great performance      |
...
| SPHINX     | YES      | Manticore storage engine                                       |
...
+------------+----------+-------------------------------------------------------------+
13 rows in set (0.00 sec)

##### Using SphinxSE¶

To search via SphinxSE, you would need to create special ENGINE=SPHINX “search table”, and then SELECT from it with full text query put into WHERE clause for query column.

Let’s begin with an example create statement and search query:

CREATE TABLE t1
(
id          INTEGER UNSIGNED NOT NULL,
weight      INTEGER NOT NULL,
query       VARCHAR(3072) NOT NULL,
group_id    INTEGER,
INDEX(query)
) ENGINE=SPHINX CONNECTION="sphinx://localhost:9312/test";

SELECT * FROM t1 WHERE query='test it;mode=any';


First 3 columns of search table must have a types of INTEGER UNSINGED or BIGINT for the 1st column (document id), INTEGER or BIGINT for the 2nd column (match weight), and VARCHAR or TEXT for the 3rd column (your query), respectively. This mapping is fixed; you can not omit any of these three required columns, or move them around, or change types. Also, query column must be indexed; all the others must be kept unindexed. Columns’ names are ignored so you can use arbitrary ones.

Additional columns must be either INTEGER, TIMESTAMP, BIGINT, VARCHAR, or FLOAT. They will be bound to attributes provided in Manticore result set by name, so their names must match attribute names specified in sphinx.conf. If there’s no such attribute name in Manticore search results, column will have NULL values.

Special “virtual” attributes names can also be bound to SphinxSE columns. _sph_ needs to be used instead of @ for that. For instance, to obtain the values of @groupby, @count, or @distinct virtual attributes, use _sph_groupby, _sph_count or _sph_distinct column names, respectively.

CONNECTION string parameter can be used to specify default searchd host, port and indexes for queries issued using this table. If no connection string is specified in CREATE TABLE, index name “*” (ie. search all indexes) and localhost:9312 are assumed. Connection string syntax is as follows:

CONNECTION="sphinx://HOST:PORT/INDEXNAME"


You can change the default connection string later:

mysql> ALTER TABLE t1 CONNECTION="sphinx://NEWHOST:NEWPORT/NEWINDEXNAME";


You can also override all these parameters per-query.

As seen in example, both query text and search options should be put into WHERE clause on search query column (ie. 3rd column); the options are separated by semicolons; and their names from values by equality sign. Any number of options can be specified. Available options are:

• query - query text;
• mode - matching mode. Must be one of “all”, “any”, “phrase”, “boolean”, or “extended”. Default is “all”;
• sort - match sorting mode. Must be one of “relevance”, “attr_desc”, “attr_asc”, “time_segments”, or “extended”. In all modes besides “relevance” attribute name (or sorting clause for “extended”) is also required after a colon:
... WHERE query='test;sort=attr_asc:group_id';
... WHERE query='test;sort=extended:@weight desc, group_id asc';

• offset - offset into result set, default is 0;
• limit - amount of matches to retrieve from result set, default is 20;
• index - names of the indexes to search:
... WHERE query='test;index=test1;';
... WHERE query='test;index=test1,test2,test3;';

• minid, maxid - min and max document ID to match;
• weights - comma-separated list of weights to be assigned to Manticore full-text fields:
... WHERE query='test;weights=1,2,3;';

• filter, !filter - comma-separated attribute name and a set of values to match:
# only include groups 1, 5 and 19
... WHERE query='test;filter=group_id,1,5,19;';

# exclude groups 3 and 11
... WHERE query='test;!filter=group_id,3,11;';

• range, !range - comma-separated (integer or bigint) Manticore attribute name, and min and max values to match:
# include groups from 3 to 7, inclusive
... WHERE query='test;range=group_id,3,7;';

# exclude groups from 5 to 25
... WHERE query='test;!range=group_id,5,25;';

• floatrange, !floatrange - comma-separated (floating point) Manticore attribute name, and min and max values to match:
# filter by a float size
... WHERE query='test;floatrange=size,2,3;';

# pick all results within 1000 meter from geoanchor
... WHERE query='test;floatrange=@geodist,0,1000;';

• maxmatches - per-query max matches value, as in max_matches parameter to SetLimits() API call:
... WHERE query='test;maxmatches=2000;';

• cutoff - maximum allowed matches, as in cutoff parameter to SetLimits() API call:
... WHERE query='test;cutoff=10000;';

• maxquerytime - maximum allowed query time (in milliseconds), as in SetMaxQueryTime() API call:
... WHERE query='test;maxquerytime=1000;';

• groupby - group-by function and attribute, corresponding to SetGroupBy() API call:
... WHERE query='test;groupby=day:published_ts;';
... WHERE query='test;groupby=attr:group_id;';

• groupsort - group-by sorting clause:
... WHERE query='test;groupsort=@count desc;';

• distinct - an attribute to compute COUNT(DISTINCT) for when doing group-by, as in SetGroupDistinct() API call:
... WHERE query='test;groupby=attr:country_id;distinct=site_id';

• indexweights - comma-separated list of index names and weights to use when searching through several indexes:
... WHERE query='test;indexweights=idx_exact,2,idx_stemmed,1;';

• fieldweights - comma-separated list of per-field weights that can be used by the ranker:
... WHERE query='test;fieldweights=title,10,abstract,3,content,1;';

• comment - a string to mark this query in query log (mapping to $comment parameter in Query() API call): ... WHERE query='test;comment=marker001;';  • select - a string with expressions to compute (mapping to SetSelect() API call): ... WHERE query='test;select=2*a+3*** as myexpr;';  • host, port - remote searchd host name and TCP port, respectively: ... WHERE query='test;host=sphinx-test.loc;port=7312;';  • ranker - a ranking function to use with “extended” matching mode, as in SetRankingMode() API call (the only mode that supports full query syntax). Known values are “proximity_bm25”, “bm25”, “none”, “wordcount”, “proximity”, “matchany”, “fieldmask”, “sph04”, “expr:EXPRESSION” syntax to support expression-based ranker (where EXPRESSION should be replaced with your specific ranking formula), and “export:EXPRESSION”: ... WHERE query='test;mode=extended;ranker=bm25;'; ... WHERE query='test;mode=extended;ranker=expr:sum(lcs);';  The “export” ranker works exactly like ranker=expr, but it stores the per-document factor values, while ranker=expr discards them after computing the final WEIGHT() value. Note that ranker=export is meant to be used but rarely, only to train a ML (machine learning) function or to define your own ranking function by hand, and never in actual production. When using this ranker, you’ll probably want to examine the output of the RANKFACTORS() function that produces a string with all the field level factors for each document.  SELECT *, WEIGHT(), RANKFACTORS() FROM myindex WHERE MATCH('dog') OPTION ranker=export('100*bm25') would produce something like  *************************** 1\. row *************************** id: 555617 published: 1110067331 channel_id: 1059819 title: 7 content: 428 weight(): 69900 rankfactors(): bm25=699, bm25a=0.666478, field_mask=2, doc_word_count=1, field1=(lcs=1, hit_count=4, word_count=1, tf_idf=1.038127, min_idf=0.259532, max_idf=0.259532, sum_idf=0.259532, min_hit_pos=120, min_best_span_pos=120, exact_hit=0, max_window_hits=1), word1=(tf=4, idf=0.259532) *************************** 2\. row *************************** id: 555313 published: 1108438365 channel_id: 1058561 title: 8 content: 249 weight(): 68500 rankfactors(): bm25=685, bm25a=0.675213, field_mask=3, doc_word_count=1, field0=(lcs=1, hit_count=1, word_count=1, tf_idf=0.259532, min_idf=0.259532, max_idf=0.259532, sum_idf=0.259532, min_hit_pos=8, min_best_span_pos=8, exact_hit=0, max_window_hits=1), field1=(lcs=1, hit_count=2, word_count=1, tf_idf=0.519063, min_idf=0.259532, max_idf=0.259532, sum_idf=0.259532, min_hit_pos=36, min_best_span_pos=36, exact_hit=0, max_window_hits=1), word1=(tf=3, idf=0.259532)  • geoanchor - geodistance anchor, as in SetGeoAnchor() API call. Takes 4 parameters which are latitude and longitude attribute names, and anchor point coordinates respectively: ... WHERE query='test;geoanchor=latattr,lonattr,0.123,0.456';  One very important note that it is much more efficient to allow Manticore to perform sorting, filtering and slicing the result set than to raise max matches count and use WHERE, ORDER BY and LIMIT clauses on MySQL side. This is for two reasons. First, Manticore does a number of optimizations and performs better than MySQL on these tasks. Second, less data would need to be packed by searchd, transferred and unpacked by SphinxSE. Additional query info besides result set could be retrieved with SHOW ENGINE SPHINX STATUS statement: mysql> SHOW ENGINE SPHINX STATUS; +--------+-------+-------------------------------------------------+ | Type | Name | Status | +--------+-------+-------------------------------------------------+ | SPHINX | stats | total: 25, total found: 25, time: 126, words: 2 | | SPHINX | words | sphinx:591:1256 soft:11076:15945 | +--------+-------+-------------------------------------------------+ 2 rows in set (0.00 sec)  This information can also be accessed through status variables. Note that this method does not require super-user privileges. mysql> SHOW STATUS LIKE 'sphinx_%'; +--------------------+----------------------------------+ | Variable_name | Value | +--------------------+----------------------------------+ | sphinx_total | 25 | | sphinx_total_found | 25 | | sphinx_time | 126 | | sphinx_word_count | 2 | | sphinx_words | sphinx:591:1256 soft:11076:15945 | +--------------------+----------------------------------+ 5 rows in set (0.00 sec)  You could perform JOINs on SphinxSE search table and tables using other engines. Here’s an example with “documents” from example.sql: mysql> SELECT content, date_added FROM test.documents docs -> JOIN t1 ON (docs.id=t1.id) -> WHERE query="one document;mode=any"; +-------------------------------------+---------------------+ | content | docdate | +-------------------------------------+---------------------+ | this is my test document number two | 2006-06-17 14:04:28 | | this is my test document number one | 2006-06-17 14:04:28 | +-------------------------------------+---------------------+ 2 rows in set (0.00 sec) mysql> SHOW ENGINE SPHINX STATUS; +--------+-------+---------------------------------------------+ | Type | Name | Status | +--------+-------+---------------------------------------------+ | SPHINX | stats | total: 2, total found: 2, time: 0, words: 2 | | SPHINX | words | one:1:2 document:2:2 | +--------+-------+---------------------------------------------+ 2 rows in set (0.00 sec)  #### Building snippets (excerpts) via MySQL¶ SphinxSE also includes a UDF function that lets you create snippets through MySQL. The functionality is fully similar to BuildExcerprts API call but accessible through MySQL+SphinxSE. The binary that provides the UDF is named sphinx.so and should be automatically built and installed to proper location along with SphinxSE itself. If it does not get installed automatically for some reason, look for sphinx.so in the build directory and copy it to the plugins directory of your MySQL instance. After that, register the UDF using the following statement: CREATE FUNCTION sphinx_snippets RETURNS STRING SONAME 'sphinx.so';  Function name must be sphinx_snippets, you can not use an arbitrary name. Function arguments are as follows: Prototype: function sphinx_snippets ( document, index, words, [options] ); Document and words arguments can be either strings or table columns. Options must be specified like this: 'value' AS option_name. For a list of supported options, refer to BuildExcerprts() API call. The only UDF-specific additional option is named sphinx and lets you specify searchd location (host and port). Usage examples: SELECT sphinx_snippets('hello world doc', 'main', 'world', 'sphinx://192.168.1.1/' AS sphinx, true AS exact_phrase, '[**]' AS before_match, '[/**]' AS after_match) FROM documents; SELECT title, sphinx_snippets(text, 'index', 'mysql php') AS text FROM sphinx, documents WHERE query='mysql php' AND sphinx.id=documents.id;  ### MySQL FEDERATED storage engine support¶ #### MySQL FEDERATED overview¶ The FEDERATED storage engine lets you access data from a remote MySQL database without using replication or cluster technology. Querying a local FEDERATED table automatically pulls the data from the remote Manticore index. No data is stored on the local tables. With FEDERATED engine a MySQL server can connect to a local or remote Manticore daemon and perform search queries. Performing queries via FEDERATED is similar to SphinxSE plugin. Unlike SphinxSE, the FEDERATED engine is bundled in all MySQL installs and can be used with Manticore out of the box, without any additional plugin compiling or changes to the MySQL server. #### Using FEDERATED¶ An actual Manticore query cannot be used directly with FEDERATED engine and must be “proxied” (send as a string in a column) due to limitations of FEDERATED engine and the fact that Manticore implements custom syntax like the MATCH clause. To search via FEDERATED, you would need to create first a FEDERATED engine table. The Manticore query will be included in a query column in the SELECT performed over the FEDERATED table. Let’s begin with an example create statement and search query: CREATE TABLE t1 ( id INTEGER UNSIGNED NOT NULL, channel_id INTEGER NOT NULL, group_id INTEGER, query VARCHAR(3072) NOT NULL, INDEX(query) ) ENGINE=FEDERATED DEFAULT CHARSET=utf8 CONNECTION='mysql://FEDERATED@127.0.0.1:9306/DB/test_index'; SELECT * FROM t1 WHERE query='SELECT * FROM test_index WHERE MATCH (\'pile box\') AND channel_id<1000 GROUP BY group_id';  The only fixed mapping is query column. It is mandatory and must be the only column with an index attached. The Manticore index that is linked via FEDERATED must be an index with storage (plain or RealTime). FEDERATED table should have columns with same names as remote Manticore index attributes as will be bound to attributes provided in Manticore result set by name, however might map not all attributes but only some of them. Arbitrary expression from query select list which name “hides” index attribute will be used at result set. Manticore daemon identifies query from FEDERATED client by user name “FEDERATED”. CONNECTION string parameter should be used to specify searchd host, SphinxQL port and indexes for queries issued using this table. Connection string syntax is as follows: CONNECTION="mysql://FEDERATED@HOST:PORT/DB/INDEXNAME"  Since Manticore doesn’t have the concept of database, the DB string can be random as it will be ignored by Manticore, but MySQL requires a value in the CONNECTION string definition. As seen in example, full SELECT SphinxQL query should be put into WHERE clause on search query column SELECT syntax. Only SELECT statement is supported but not INSERT, REPLACE, UPDATE, DELETE. One very important note that it is much more efficient to allow Manticore to perform sorting, filtering and slicing the result set than to raise max matches count and use WHERE, ORDER BY and LIMIT clauses on MySQL side. This is for two reasons. First, Manticore does a number of optimizations and performs better than MySQL on these tasks. Second, less data would need to be packed by searchd, transferred and unpacked between Manticore and MySQL. JOINs can be performed between FEDERATED table and other MySQL tables. This can be used to retrieve information that is not stored in the Manticore index. mysql> select t1.id,mysqltable.longtext from t1 join mysqltable on t1.id=mysqltable.id where query='SELECT * from test_index where match(\'pile box\')'  ### Percolate query¶ The percolate query is used to match documents against queries stored in an index. It is also called “search in reverse” as it works opposite to a regular search where documents are stored in an index and queries are issued against the index. Queries are stored in special kind index and they can be added, deleted and listed using INSERT/DELETE/SELECT statements similar way as it’s done for a regular index. Checking if a document matches any of the predefined criterias (queries) performed via sphinxql CALL PQ function, or via http /json/pq/<index>/_search endpoint. They returns list of matched queries and may be additional info as matching clause, filters, and tags. The workflow of using percolate queries is the following: • a percolate index is defined in the configuration • queries are inserted in the percolate index in same way as documents for a Real-Time index • documents can be tested against the stored queries with CALL PQ statement #### The percolate index¶ A percolate query works only for percolate index type. The percolate index internaly is a modified Real-Time index and shares a similar configuration. index pq { type = percolate path = path/index_name min_infix_len = 4 rt_field = title rt_field = body rt_attr_uint = author_id }  The fields and attributes declared in the configuration define the document schema used by percolate queries and documents that will be tested against stored queries using CALL PQ command. The schema can be viewed with DESCRIBE table_name TABLE statement. If the schema is omited, index will imply default field text and an integer attribute gid. The default field and attribute are removed when defining explicit a schema, but their name can be reused if needed. The stored queries and the tested documents must respect the defined schema in the percolate index configuration. Trying to use fields/attributes not declared in the schema will give results with no matches. The schema of the percolate index itself which can be viewed with DESCRIBE table_name and which will be used in INSERT statements to add queries in the percolate index uses a fixed structure and contain the following elements: ##### ID¶ The “id” is an unsigned 64-bit integer with autoincrement functionality therefor it can be ommited in INSERT statements. ##### Query¶ It holds the full text query as the value of a MATCH clause. If per field operators are used inside a query, the full text fields needs to be declared in the percolate index configuration. If the stored query is supposed to be a full-scan (only attribute filtering, no full text query), the query value can be empty or simply omitted. ##### Filters¶ Filters is an optional string containing attribute filters and/or expressions in the same way they are defined in the WHERE clause, like gid=10 and pages>4. The attributes used here needs to be declared in the percolate index configuration. ##### Tags¶ Optional, tags represent a list of string labels, separated by comma, which can be used for filtering queries in SELECT statements on the percolate index or to delete queries using DELETE statement. The tags can be returned in the CALL PQ result set. #### Index schemas¶ Usual sphinxql command DESCRIBE will reveal you both internal (schema for call pq) and external (schema for select). #### Store queries¶ Storing queries is done either via usual INSERT statement, either via http json/pq/<idx>/doc endpoint. Read appropriate sections for syntax details and features. #### List stored queries¶ To list stored queries either use SELECT statement, or http json/search endpoint. (endpoint /json/pq/<index>/search is deprecated and will be removed). From viewpoint of these methods just know that percolate index doesn’t contains any full-text fields, so match() clause will not work. You have just an id and few columns to operate:  Field Type id bigint query string tags string filters string So, you can fire any usual full-scan queries, like SELECT * FROM pq; SELECT * FROM pq WHERE tags='tags list'; SELECT * FROM pq WHERE id IN (11,35,101); SELECT * FROM pq WHERE tags ANY ('foo', 'bar'); SELECT * FROM pq WHERE tags NOT ANY ('foo', 'bar'); SELECT * FROM pq WHERE tags ALL ('foo', 'bar'); SELECT * FROM pq WHERE tags NOT ALL ('foo', 'bar'); SELECT * FROM pq LIMIT 1300, 45; SELECT * FROM distributed_pq LIMIT 5;  #### Delete queries¶ To delete a stored percolate query(es) in index either use DELETE statement, http json/delete endpoint. (also endpoint /json/pq/<index>/delete works, but avoid to use it) DELETE FROM index_name WHERE id=1; DELETE FROM index_name WHERE tags ANY ('tags', 'list');  TRUNCATE RTINDEX statement can also be used to delete all stored queries: TRUNCATE RTINDEX index_name;  #### Search matching queries¶ That is main purpose of percolate indexes. You provide one or many documents according to internal schema, defined in config, and percolate index gives you matched queries. It may be done either by CALL PQ statement in sphinxql, or by using http json/pq/pq_index/_search endpoint. To search for queries matching a document(s) the CALL PQ statement is used which looks like CALL PQ ('index_name', 'single document', 0 AS docs, 0 AS docs_json, 0 AS verbose); CALL PQ ('index_name', ('multiple documents', 'go this way'), 0 AS docs_json );  Or via http POST json/pq/idx_pq_1/_search { "query": { "percolate": { "document" : { "title" : "some text to match" } } } }  Searching for matching queries performance is affected by dist_threads. It transparently works with distributed percolate indexes. #### Meta¶ Meta information is kept for documents on “CALL PQ” and can be retrieved with SHOW META call. SHOW META output after CALL PQ looks like +-------------------------+-----------+ | Name | Value | +-------------------------+-----------+ | Total | 0.010 sec | | Queries matched | 950 | | Document matched | 1500 | | Total queries stored | 1000 | | Term only queries | 998 | +-------------------------+-----------+  With entries: • Total - total time spent for matching the document(s) • Queries matched - how many stored queries match the document(s) • Document matches - how many times the documents match the queries stored in the index • Total queries stored - how many queries are stored in the index at all • Term only queries - how many queries in the index have terms. The rest of the queries have extended query syntax If you used option verbose when invoking CALL PQ, output will be more detailed: +-------------------------+-----------+ | Name | Value | +-------------------------+-----------+ | Total | 0.000 sec | | Setup | 0.000 sec | | Queries matched | 2 | | Queries failed | 0 | | Document matched | 2 | | Total queries stored | 5 | | Term only queries | 5 | | Fast rejected queries | 3 | | Time per query | 93, 30 | | Time of matched queries | 123 | +-------------------------+-----------+  Here you see additional entries: • Setup - time spent to initial setup of matching process - parsing docs, setting options, etc. • Queries failed - number of failed queries • Fast rejected queries - num of queries which wasn’t fall into full routine, but quickly matched and rejected with filters or other conditions • Time per query - detailed times per each query • Time of matched queries - total time spend to matched queries #### Reconfigure¶ As well as for RealTime indexes ALTER RECONFIGURE command is also supported for percolate query index. It allows to reconfigure percolate index on the fly without deleting and repopulating the index with queries back. mysql> DESC pq1; +-------+--------+ | Field | Type | +-------+--------+ | id | bigint | | text | field | | body | field | | k | uint | +-------+--------+ mysql> SELECT * FROM pq1; +------+-------+------+-------------+ | UID | Query | Tags | Filters | +------+-------+------+-------------+ | 1 | test | | k=4 | | 2 | test | | k IN (4,6) | | 3 | test | | | +------+-------+------+-------------+  Add JSON attribute to the index config rt_attr_json = json_data, then issue ALTER RECONFIGURE mysql> ALTER RTINDEX pq1 RECONFIGURE; mysql> DESC pq1; +-----------+--------+ | Field | Type | +-----------+--------+ | id | bigint | | text | field | | body | field | | k | uint | | json_data | json | +-----------+--------+  #### Distributed indexes made from percolate locals and/or agents (DPQ indexes)¶ You can construct a distributed index from several percolate indexes. The syntax is absolutely the same as for other distributed indexes. It can include several local indexes as well as several agents. For local the only noticeable difference is that since percolate indexes don’t know about kill-lists there’s no difference in which order they’re mentioned in a distributed index definition. For DPQ the operations of listing stored queries and searching through them (CALL PQ) are transparent and works as if all the indexes were one solid local index. However data manipulation statements such as insert, replace, truncate are not available. If you mention a non-pq index among the agents, the behaviour will be undefined. Most likely in case if the erroneous agent has the same schema as the outer schema of the pq index (id, query, tags, filters) - it will not trigger an error when listing stored PQ rules hence may pollute the list of actual PQ rules stored in PQ indexes with it’s own non-pq strings, so be aware of the confusion! ‘CALL PQ’ to such wrong agent will definitely trigger an error. ## Extending¶ ### UDFs (User Defined Functions)¶ Our expression engine can be extended with user defined functions, or UDFs for short, like this: SELECT id, attr1, myudf(attr2, attr3+attr4) ...  You can load and unload UDFs dynamically into searchd without having to restart the daemon, and used them in expressions when searching, ranking, etc. Quick summary of the UDF features is as follows. • UDFs can take integer (both 32-bit and 64-bit), float, string, MVA, or PACKEDFACTORS() arguments. • UDFs can return integer, float, or string values. • UDFs can check the argument number, types, and names during the query setup phase, and raise errors. • Aggregation UDFs are not yet supported (but might be in the future). UDFs have a wide variety of uses, for instance: • adding custom mathematical or string functions; • accessing the database or files from within Manticore; • implementing complex ranking functions. UDFs reside in the external dynamic libraries (.so files on UNIX and .dll on Windows systems). Library files need to reside in a trusted folder specified by plugin_dir directive, for obvious security reasons: securing a single folder is easy; letting anyone install arbitrary code into searchd is a risk. You can load and unload them dynamically into searchd with CREATE FUNCTION and DROP FUNCTION SphinxQL statements respectively. Also, you can seamlessly reload UDFs (and other plugins) with RELOAD PLUGINS statement. Manticore keeps track of the currently loaded functions, that is, every time you create or drop an UDF, searchd writes its state to the sphinxql_state file as a plain good old SQL script. Once you successfully load an UDF, you can use it in your SELECT or other statements just as well as any of the builtin functions: SELECT id, MYCUSTOMFUNC(groupid, authorname), ... FROM myindex  Multiple UDFs (and other plugins) may reside in a single library. That library will only be loaded once. It gets automatically unloaded once all the UDFs and plugins from it are dropped. In theory you can write an UDF in any language as long as its compiler is able to import standard C header, and emit standard dynamic libraries with properly exported functions. Of course, the path of least resistance is to write in either C++ or plain C. We provide an example UDF library written in plain C and implementing several functions (demonstrating a few different techniques) along with our source code, see src/udfexample.c. That example includes src/sphinxudf.h header file definitions of a few UDF related structures and types. For most UDFs and plugins, a mere #include "sphinxudf.h", like in the example, should be completely sufficient, too. However, if you’re writing a ranking function and need to access the ranking signals (factors) data from within the UDF, you will also need to compile and link with src/sphinxudf.c (also available in our source code), because the implementations of the fuctions that let you access the signal data from within the UDF reside in that file. Both sphinxudf.h header and sphinxudf.c are standalone. So you can copy around those files only; they do not depend on any other bits of Manticore source code. Within your UDF, you must implement and export only a couple functions, literally. First, for UDF interface version control, you must define a function int LIBRARYNAME_ver(), where LIBRARYNAME is the name of your library file, and you must return SPH_UDF_VERSION (a value defined in sphinxudf.h) from it. Here’s an example. #include <sphinxudf.h> // our library will be called udfexample.so, thus, so it must define // a version function named udfexample_ver() int udfexample_ver() { return SPH_UDF_VERSION; }  That protects you from accidentally loading a library with a mismatching UDF interface version into a newer or older searchd. Second, yout must implement the actual function, too. sphinx_int64_t testfunc ( SPH_UDF_INIT * init, SPH_UDF_ARGS * args, char * error_flag ) { return 123; } UDF function names in SphinxQL are case insensitive. However, the respective C function names are not, they need to be all lower-case, or the UDF will not load. More importantly, it is vital that a) the calling convention is C (aka __cdecl), b) arguments list matches the plugin system expectations exactly, and c) the return type matches the one you specify in CREATE FUNCTION. Unfortunately, there is no (easy) way for us to check for those mistakes when loading the function, and they could crash the server and/or result in unexpected results. Last but not least, all the C functions you implement need to be thread-safe. The first argument, a pointer to SPH_UDF_INIT structure, is essentially a pointer to our function state. It is option. In the example just above the function is stateless, it simply returns 123 every time it gets called. So we do not have to define an initialization function, and we can simply ignore that argument. The second argument, a pointer to SPH_UDF_ARGS, is the most important one. All the actual call arguments are passed to your UDF via this structure; it contians the call argument count, names, types, etc. So whether your function gets called like SELECT id, testfunc(1) or like SELECT id, testfunc('abc', 1000*id+gid, WEIGHT()) or anyhow else, it will receive the very same SPH_UDF_ARGS structure in all of these cases. However, the data passed in the args structure will be different. In the first example args->arg_count will be set to 1, in the second example it will be set to 3, args->arg_types array will contain different type data, and so on. Finally, the third argument is an error flag. UDF can raise it to indicate that some kinda of an internal error happened, the UDF can not continue, and the query should terminate early. You should not use this for argument type checks or for any other error reporting that is likely to happen during normal use. This flag is designed to report sudden critical runtime errors, such as running out of memory. If we wanted to, say, allocate temporary storage for our function to use, or check upfront whether the arguments are of the supported types, then we would need to add two more functions, with UDF initialization and deinitialization, respectively. int testfunc_init ( SPH_UDF_INIT * init, SPH_UDF_ARGS * args, char * error_message ) { // allocate and initialize a little bit of temporary storage init->func_data = malloc ( sizeof(int) ); *(int*)init->func_data = 123; // return a success code return 0; } void testfunc_deinit ( SPH_UDF_INIT * init ) { // free up our temporary storage free ( init->func_data ); }  Note how testfunc_init() also receives the call arguments structure. By the time it is called it does not receive any actual values, so the args->arg_values will be NULL. But the argument names and types are known and will be passed. You can check them in the initialization function and return an error if they are of an unsupported type. UDFs can receive arguments of pretty much any valid internal Manticore type. Refer to sphinx_udf_argtype enumeration in sphinxudf.h for a full list. Most of the types map straightforwardly to the respective C types. The most notable exception is the SPH_UDF_TYPE_FACTORS argument type. You get that type by calling your UDF with a PACKEDFACTOR() argument. It’s data is a binary blob in a certain internal format, and to extract individual ranking signals from that blob, you need to use either of the two sphinx_factors_XXX() or sphinx_get_YYY_factor() families of functions. The first family consists of just 3 functions, sphinx_factors_init() that initializes the unpacked SPH_UDF_FACTORS structure, sphinx_factors_unpack() that unpacks a binary blob into it, and sphinx_factors_deinit() that cleans up an deallocates the SPH_UDF_FACTORS. So you need to call init() and unpack(), then you can use the SPH_UDF_FACTORS fields, and then you need to cleanup with deinit(). That is simple, but results in a bunch of memory allocations per each processed document, and might be slow. The other interface, consisting of a bunch of sphinx_get_YYY_factor() functions, is a little more wordy to use, but accesses the blob data directly and guarantees that there will be zero allocations. So for top-notch ranking UDF performance, you want to use that one. As for the return types, UDFs can currently return a signle INTEGER, BIGINT, FLOAT, or STRING value. The C function return type should be sphinx_int64_t, sphinx_int64_t, double, or char* respectively. In the last case you must use args->fn_malloc function to allocate the returned string values. Internally in your UDF you can use whatever you want, so the testfunc_init() example above is correct code even though it uses malloc() directly: you manage that pointer yourself, it gets freed up using a matching free() call, and all is well. However, the returned strings values are managed by Manticore and we have our own allocator, so for the return values specifically, you need to use it too. Depending on how your UDFs are used in the query, the main function call (testfunc() in our example) might be called in a rather different volume and order. Specifically, • UDFs referenced in WHERE, ORDER BY, or GROUP BY clauses must and will be evaluated for every matched document. They will be called in the natural matching order. • without subselects, UDFs that can be evaluated at the very last stage over the final result set will be evaluated that way, but before applying the LIMIT clause. They will be called in the result set order. • with subselects, such UDFs will also be evaluated after applying the inner LIMIT clause. The calling sequence of the other functions is fixed, though. Namely, • testfunc_init() is called once when initializing the query. It can return a non-zero code to indicate a failure; in that case query will be terminated, and the error message from the error_message buffer will be returned. • testfunc() is called for every eligible row (see above), whenever Manticore needs to compute the UDF value. It can also indicate an (internal) failure error by writing a non-zero byte value to error_flag. In that case, it is guaranteed that will no more be called for subsequent rows, and a default return value of 0 will be substituted. Manticore might or might not choose to terminate such queries early, neither behavior is currently guaranteed. • testfunc_deinit() is called once when the query processing (in a given index shard) ends. We do not yet support aggregation functions. In other words, your UDFs will be called for just a single document at a time and are expected to return some value for that document. Writing a function that can compute an aggregate value like AVG() over the entire group of documents that share the same GROUP BY key is not yet possible. However, you can use UDFs within the builtin aggregate functions: that is, even though MYCUSTOMAVG() is not supported yet, AVG(MYCUSTOMFUNC()) should work alright! UDFs are local. In order to use them on a cluster, you have to put the same library on all its nodes and run CREATEs on all the nodes too. This might change in the future versions. ### Plugins¶ Here’s the complete plugin type list. • UDF plugins; • ranker plugins; • indexing-time token filter plugins; • query-time token filter plugins. This section discusses writing and managing plugins in general; things specific to writing this or that type of a plugin are then discussed in their respective subsections. So, how do you write and use a plugin? Four-line crash course goes as follows: • create a dynamic library (either .so or.dll), most likely in C or C++; • load that plugin into searchd using CREATE PLUGIN; • invoke it using the plugin specific calls (typically using this or that OPTION). • to unload or reload a plugin use DROP PLUGIN and RELOAD PLUGINS respectively. Note that while UDFs are first-class plugins they are nevertheless installed using a separate CREATE FUNCTION statement. It lets you specify the return type neatly so there was especially little reason to ruin backwards compatibility and change the syntax. Dynamic plugins are supported in threads and thread_pool workers. Multiple plugins (and/or UDFs) may reside in a single library file. So you might choose to either put all your project-specific plugins in a single common big library; or you might choose to have a separate library for every UDF and plugin; that is up to you. Just as with UDFs, you want to include src/sphinxudf.h header file. At the very least, you will need the SPH_UDF_VERSION constant to implement a proper version function. Depending on the specific plugin type, you might or might not need to link your plugin with src/sphinxudf.c. However, all the functions implemented in sphinxudf.c are about unpacking the PACKEDFACTORS() blob, and no plugin types are exposed to that kind of data. So currently, you would never need to link with the C-file, just the header would be sufficient. (In fact, if you copy over the UDF version number, then for some of the plugin types you would not even need the header file.) Formally, plugins are just sets of C functions that follow a certain naming parttern. You are typically required to define just one key function that does the most important work, but you may define a bunch of other functions, too. For example, to implement a ranker called “myrank”, you must define myrank_finalize() function that actually returns the rank value, however, you might also define myrank_init(), myrank_update(), and myrank_deinit() functions. Specific sets of well-known suffixes and the call arguments do differ based on the plugin type, but _init() and _deinit() are generic, every plugin has those. Protip: for a quick reference on the known suffixes and their argument types, refer to sphinxplugin.h, we define the call prototoypes in the very beginning of that file. Despite having the public interface defined in ye good olde good pure C, our plugins essentially follow the object-oriented model. Indeed, every _init() function receives a void ** userdata out-parameter. And the pointer value that you store at (*userdata) location is then be passed as a 1st argument to all the other plugin functions. So you can think of a plugin as class that gets instantiated every time an object of that class is needed to handle a request: the userdata pointer would be its this pointer; the functions would be its methods, and the _init() and _deinit() functions would be the constructor and destructor respectively. Why this (minor) OOP-in-C complication? Well, plugins run in a multi-threaded environment, and some of them have to be stateful. You can’t keep that state in a global variable in your plugin. So we have to pass around a userdata parameter anyway to let you keep that state. And that naturally brings us to the OOP model. And if you’ve got a simple, stateless plugin, the interface lets you omit the _init() and _deinit() and whatever other functions just as well. To summarize, here goes the simplest complete ranker plugin, in just 3 lines of C code. // gcc -fPIC -shared -o myrank.so myrank.c #include "sphinxudf.h" int myrank_ver() { return SPH_UDF_VERSION; } int myrank_finalize(void *u, int w) { return 123; }  And this is how you use it: mysql> CREATE PLUGIN myrank TYPE 'ranker' SONAME 'myrank.dll'; Query OK, 0 rows affected (0.00 sec) mysql> SELECT id, weight() FROM test1 WHERE MATCH('test') -> OPTION ranker=myrank(''); +------+----------+ | id | weight() | +------+----------+ | 1 | 123 | | 2 | 123 | +------+----------+ 2 rows in set (0.01 sec)  ### Ranker plugins¶ Ranker plugins let you implement a custom ranker that receives all the occurrences of the keywords matched in the document, and computes a WEIGHT() value. They can be called as follows: SELECT id, attr1 FROM test WHERE match('hello') OPTION ranker=myranker('option1=1');  The call workflow is as follows: 1. XXX_init() gets called once per query per index, in the very beginning. A few query-wide options are passed to it through a SPH_RANKER_INIT structure, including the user options strings (in the example just above, “option1=1” is that string). 2. XXX_update() gets called multiple times per matched document, with every matched keyword occurrence passed as its parameter, a SPH_RANKER_HIT structure. The occurrences within each document are guaranteed to be passed in the order of ascending hit->hit_pos values. 3. XXX_finalize() gets called once per matched document, once there are no more keyword occurrences. It must return the WEIGHT() value. This is the only mandatory function. 4. XXX_deinit() gets called once per query, in the very end. ### Token filter plugins¶ Token filter plugins let you implement a custom tokenizer that makes tokens according to custom rules. There are two type: Token filters processing tokens after base tokenizer processed text at field or query and made tokens from it. In the text processing pipeline, the token filters will run after the base tokenizer processing occurs (which process the text from field or query and create tokens out of them). #### Index-time tokenizer¶ Index-time tokenizer gets created by indexer on indexing source data into index or by RT index on processing INSERT or REPLACE statements. Plugin is declared as library name:plugin name:optional string of settings. The init functions of the plugin can accept arbitrary settings that can be passed as a string in format option1=value1;option2=value2;... Example: index_token_filter = my_lib.so:email_process:field=email;split=.io  The call workflow for index-time token filter is as follows: 1. XXX_init() gets called right after indexer creates token filter with empty fields list then after indexer got index schema with actual fields list. It must return zero for successful initialization or error description otherwise. 2. XXX_begin_document gets called only for RT index INSERT/REPLACE for every document. It must return zero for successful call or error description otherwise. Using OPTION token_filter_options additional parameters/settings can be passed to the function. INSERT INTO rt (id, title) VALUES (1, 'some text corp@space.io') OPTION token_filter_options='.io'  3. XXX_begin_field gets called once for each field prior to processing field with base tokenizer with field number as its parameter. 4. XXX_push_token gets called once for each new token produced by base tokenizer with source token as its parameter. It must return token, count of extra tokens made by token filter and delta position for token. 5. XXX_get_extra_token gets called multiple times in case XXX_push_token reports extra tokens. It must return token and delta position for that extra token. 6. XXX_end_field gets called once right after source tokens from current field get over. 7. XXX_deinit gets called in the very end of indexing. The following functions are mandatory to be defined: XXX_begin_document and XXX_push_token and XXX_get_extra_token. #### query-time token filter¶ Query-time tokenizer gets created on search each time full-text invoked by every index involved. The call workflow for query-time token filter is as follows: 1. XXX_init() gets called once per index prior to parsing query with parameters - max token length and string set by token_filter option SELECT * FROM index WHERE MATCH ('test') OPTION token_filter='my_lib.so:query_email_process:io'  It must return zero for successful initialization or error description otherwise. 2. XXX_push_token() gets called once for each new token produced by base tokenizer with parameters: token produced by base tokenizer, pointer to raw token at source query string and raw token length. It must return token and delta position for token. 3. XXX_pre_morph() gets called once for token right before it got passed to morphology processor with reference to token and stopword flag. It might set stopword flag to mark token as stopword. 4. XXX_post_morph() gets called once for token after it processed by morphology processor with reference to token and stopword flag. It might set stopword flag to mark token as stopword. It must return flag non-zero value of which means to use token prior to morphology processing. 5. XXX_deinit() gets called in the very end of query processing. Absence of any of the functions is tolerated. ## Replication¶ Warning Please note that this feature is in preview stage. Some functionality may be not yet complete and may suffer changes. Read carefully changelogs of future updates to avoid possible breakages. Manticore search daemon can replicate a write transaction (INSERT, REPLACE, DELETE, TRUNCATE, etc) in an index to other nodes in the cluster. Currently only percolate indexes are supported, rt index support is in progress. Only Linux packages and builds support replication, Windows and MacOS packages do not support replication. We took advantage of Percona’s fork of Galera library which gives the following benefits: • true multi-master - read and write to any node at any time • synchronous replication - no slave lag, no data is lost after a node crash • hot standby - no downtime during failover (since there is no failover) • tightly coupled - all the nodes hold the same state. No diverged data between nodes allowed • automatic node provisioning - no need to manually back up the database and restore it on a new node • easy to use and deploy • detection and automatic eviction of unreliable nodes • certification based replication To use replication in Manticore search: • the daemon should be built with replication support (enabled in the builds Manticore provides) • data_dir option should be set in searchd section of config • there should be a listen for the replication protocol directive containing an external IP address that should not be equal to 0.0.0.0 along with a ports range defined and these “address - port range” pairs should be different for all the daemons on the same box. As a rule of thumb, port range should specify no less than two ports per cluster. • there should be at least one value of listen for the SphinxAPI protocol directive ### Replication cluster¶ Replication cluster is a set of nodes among which a write transaction gets replicated. Replication is configured on the per-index basis. One index can be assigned to only one cluster. There is no restriction on how many indexes a cluster may have. All transactions such as INSERT, REPLACE, DELETE, TRUNCATE in any percolate index belonging to a cluster are replicated to all the other nodes in the cluster. Replication is multi-master, so writes to any particular node or to multiple nodes simultaneously work equally well. Replication cluster configuration options are: #### name¶ Specifies a name for the cluster. Should be unique. #### path¶ Data directory for a write-set cache replication and incoming indexes from other nodes. Should be unique among the other clusters in the node. Default is data_dir. #### nodes¶ A list of address:port pairs for all the nodes in the cluster (comma separated). A node’s API interface should be used for this option. It can contain the current node’s address too. This list is used to join a node to the cluster and rejoin it after restart. #### options¶ Options passed directly to Galera replication plugin as described here Galera Documentation Parameters ### Creating a cluster¶ To create a cluster you should set at least its name. In case of a single cluster or if the cluster you are creating is the first one path option may be omitted, in this case data_dir option will be used as the cluster path. For all subsequent clusters you need to specify path and this path should be available. nodes option may be also set to enumerate all the nodes in the cluster. CREATE CLUSTER posts CREATE CLUSTER click_query '/var/data/click_query/' as path CREATE CLUSTER click_query '/var/data/click_query/' as path, 'clicks_mirror1:9312,clicks_mirror2:9312,clicks_mirror3:9312' as nodes  If a cluster is created without the nodes option, the first node that gets joined to the cluster will be saved as nodes. ### Joining a cluster¶ To join an existing cluster name and any working node should be set. In case of a single cluster path might be omitted, data_dir will be used as the cluster path. For all subsequent clusters path needs to be set and it should be available. JOIN CLUSTER posts AT '10.12.1.35:9312'  A node joins a cluster by getting the data from the node provided and, if successful, it updates node lists in all the other cluster nodes similar to ALTER CLUSTER … UPDATE nodes. These lists are used to rejoin nodes to the cluster on restart. There are two lists of nodes. One is used to rejoin nodes to the cluster on restart, it is updated across all nodes by ALTER CLUSTER … UPDATE nodes. JOIN CLUSTER ... AT does the same update automatically. SHOW STATUS shows this list as cluster_post_nodes_set. The second list is a list of all active nodes used for replication. This list doesn’t require manual management. ALTER CLUSTER … UPDATE nodes actually copies this list of nodes to the list of nodes used to rejoin on restart. SHOW STATUS shows this list as cluster_post_nodes_view. When nodes are located at different network segments or in different datacenters nodes option may be set explicitly. That allows to minimize traffic between nodes and to use gateway nodes for datacenters intercommunication. The following command joins an existing cluster using the nodes option. JOIN CLUSTER click_query 'clicks_mirror1:9312;clicks_mirror2:9312;clicks_mirror3:9312' as nodes, '/var/data/click_query/' as path  Note that when this syntax is used, cluster_post_nodes_set list is not updated automatically. Use ALTER CLUSTER … UPDATE nodes to update it. JOIN CLUSTER statement completes when a node receives all the neccesary data to be in sync with all the other nodes in the cluster. ### Deleting a cluster¶ Delete statement removes a cluster specified with name. The cluster gets removed from all the nodes, but its indexes are left intact and become active local non-replicated indexes. DELETE CLUSTER click_query  ### Managing indexes¶ ALTER CLUSTER <cluster_name> ADD <index_name> adds an existing local PQ index to the cluster. The node which receives the ALTER query sends the index to the other nodes in the cluster. All the local indexes with the same name on the other nodes of the cluster get replaced with the new index. ALTER CLUSTER <cluster_name> DROP <index_name> forgets about a local PQ index, i.e., it doesn’t remove the index files on the nodes but just makes it an active non-replicated index. ALTER CLUSTER click_query ADD clicks_daily_index ALTER CLUSTER posts DROP weekly_index  ### Managing nodes¶ ALTER CLUSTER <cluster_name> UPDATE nodes statement updates node lists on each node of the cluster to include every active node in the cluster. See Joining a cluster for more info on node lists. ALTER CLUSTER posts UPDATE nodes  For example, when the cluster was initially created, the list of nodes used for rejoining the cluster was 10.10.0.1:9312,10.10.1.1:9312. Since then other nodes joined the cluster and now we have the following active nodes: 10.10.0.1:9312,10.10.1.1:9312,10.15.0.1:9312,10.15.0.3:9312. But the list of nodes used for rejoining the cluster is still the same. Running the ALTER CLUSTER ... UPDATE nodes statement copies the list of active nodes to the list of nodes used to rejoin on restart. After this, the list of nodes used on restart includes all the active nodes in the cluster. Both lists of nodes can be viewed using SHOW STATUS statement (cluster_post_nodes_set and cluster_post_nodes_view). ### Write statements¶ All write statements such as INSERT, REPLACE, DELETE, TRUNCATE that change the content of a cluster’s index should use cluster_name:index_name expression in place of an index name to make sure the change is propagated to all replicas in the cluster. An error will be triggered otherwise. INSERT INTO posts:weekly_index VALUES ( 'iphone case' ) TRUNCATE RTINDEX click_query:weekly_index  Read statements such as CALL PQ, SELECT or DESCRIBE can use either regular index names not prepended with a cluster name or cluster_name:index_name. cluster_name:index_name syntax ignores the cluster name and may be used on an index that doesn’t belong to the cluster. SELECT * FROM weekly_index CALL PQ('posts:weekly_index', 'document is here')  Insertion of a percolate query performed at multiple nodes of the same cluster at the same time with auto generated document id may trigger an error as, for now, id auto generation takes into account only the local index. This may generate a duplicate id and replication requires no id conflicts. If an insert fails, retry should work well in most cases, but it depends on the insertion rate. However, replacement of percolate queries at multiple nodes at the same time with auto generated document id may cause only the query from the last finished request to be replaced. In the future, this behavior will be improved by switching to UUIDs. ### Cluster status¶ SHOW STATUS outputs, among other information, cluster status variables. The output format is cluster_name_variable_name variable_value. Most of them are described in Galera Documentation Status. Additionally we display: • cluster_name - name of the cluster • node_state - current state of the node: closed, destroyed, joining, donor, synced • indexes_count - number of indexes managed by the cluster • indexes - list of index names managed by the cluster • nodes_set - list of nodes in the cluster defined with cluster CREATE, JOIN or ALTER UPDATE commands • nodes_view - actual list of nodes in cluster which this node sees mysql> SHOW STATUS; +----------------------------+-------------------------------------------------------------------------------------+ | Counter | Value | +----------------------------+-------------------------------------------------------------------------------------+ | cluster_name | post | | cluster_post_state_uuid | fba97c45-36df-11e9-a84e-eb09d14b8ea7 | | cluster_post_conf_id | 1 | | cluster_post_status | primary | | cluster_post_size | 5 | | cluster_post_local_index | 0 | | cluster_post_node_state | synced | | cluster_post_indexes_count | 2 | | cluster_post_indexes | pq1,pq_posts | | cluster_post_nodes_set | 10.10.0.1:9312 | | cluster_post_nodes_view | 10.10.0.1:9312,10.10.0.1:9320:replication,10.10.1.1:9312,10.10.1.1:9320:replication |  ### Cluster parameters¶ Replication plugin options can be changed using SET statement: SET CLUSTER click_query GLOBAL 'pc.bootstrap' = 1  See Galera Documentation Parameters for a list of available options. ### Restarting a cluster¶ A replication cluster requires its single node to be started as a reference point before all the other nodes join it and form a cluster. This is called cluster bootstrapping which introduces a primary component before others see that as a reference point to sync up the data from. The restart of a single node or reconnecting from a node after a shutdown can be done as usual. After the whole cluster shutdown the daemon that was stopped last should be started first with --new-cluster command line option. To make sure that the daemon is able to start as a reference point the grastate.dat file located at the cluster path should be updated with the value of 1 for safe_to_bootstrap option. I.e., both conditions, --new-cluster and safe_to_bootstrap=1, must be satisfied. An attempt to start any other node without these options set will trigger an error. To override this protection and start cluster from another daemon forcibly, --new-cluster-force command line option may be used. In case of a hard crash or an unclean shutdown of all the daemons in the cluster you need to identify the most advanced node that has the largest seqno in the grastate.dat file located at the cluster path and start that daemon with the command line key --new-cluster-force. ### Cluster with diverged nodes¶ Sometimes replicated nodes can diverge from each other. The state of all the nodes might turn into non-primary due to a network split between nodes, a cluster crash, or if the replication plugin hits an exception when determining the primary component. Then it’s necessary to select a node and promote it to the primary component. To determine which node needs to be a reference, compare the last_committed cluster status variable value on all nodes. If all the daemons are already running there’s no need to start the cluster again. You just need to promote the most advanced node to the primary component with SET statement: SET CLUSTER posts GLOBAL 'pc.bootstrap' = 1  All other nodes will reconnect to the node and resync their data based on this node. #### How to recover a cluster¶ There are cases of multiple possible scenarios where searchd service got stopped with no node in the cluster being able to serve requests. In these cases someone needs to recover the cluster or part of it. Due to multi-master nature of Galera library used for replication, a cluster is like one logical entity and takes care about each its node and node’s data consistency and keeps a cluster’s status as a whole. This allows safe writes on multiple nodes at the same time and maintains cluster integrity unlike a traditional asynchronous replication. Here we take a cluster of nodes A, B and C as an example and consider scenarios where some or all nodes are out of service and what to do to bring them back. ##### Case 1¶ Node A is stopped as usual. The other nodes receive “normal shutdown” message from node A. The cluster size is reduced and a quorum re-calculation is issued. After node A got started as usual, it joins the cluster nodes. Node A will not serve any write transaction until the join is finished and it’s fully synchronized with the cluster. If a writeset cache on a donor node B or C, which is set with a Galera cluster’s option gcache.size, still has all transactions missed at node A, node A will receive a fast incremental state transfer (IST), that is, a transfer of only missed transactions. Otherwise, a snapshot state transfer (SST) will start, that is, a transfer of index files. ##### Case 2¶ Nodes A and B are stopped as usual. That is the same situation as in previous case but the cluster’s size is reduced to 1 and node C itself forms primary component that allows it to handle write transactions. Nodes A and B may be started as usual and will join the cluster after the start. Node C becomes a “donor” and provides the transfer of the state to nodes A and B. ##### Case 3¶ All nodes are stopped as usual and the cluster is off. The problem now is how to initialize cluster and it is important that on a clean shutdown of searchd nodes write the number of the last executed transaction into the cluster directory grastate.dat file along with the safe_to_bootstrap key. The node stopped last will have the safe_to_bootstrap: 1 option and the most advanced seqno number. It is important that this node should start first to form a cluster. To bootstrap, a cluster daemon should be started on this node with the –new-cluster command line key. If another node starts first and bootstraps the cluster, then the most advanced node joins that cluster, performs full SST and receives an index file where some transactions are missed in comparison with index files it got before. That is why it is important to start the node that shut down last or to look at the cluster directory grastate.dat file to find the node with the safe_to_bootstrap: 1 option. ##### Case 4¶ Node A disappears from the cluster due to crash or network failure. Nodes B and C try to reconnect to missed node A and after failure remove node A from the cluster. The cluster quorum is valid as 2 out of 3 nodes are running and the cluster works as usual. After node A restarts it will join the cluster automatically the same way as in case 1. ##### Case 5¶ Nodes A and B disappear. Node C is not able to form the quorum alone as 1 node is less than 1.5 (half of 3). So the cluster on node C is switched to non-primary state and node C rejects any write transactions with an error message. Meanwhile, the single node C is waiting for other nodes to connect and try to connect them itself. If this happens, after the network is restored and nodes A and B are running again, the cluster will be formed again automatically. If nodes A and B are just cut from node C, but they can still reach each other, they keep working as usual because they still form the quorum. However, if both nodes A and B crashed or restarted due to power outage, someone should turn on primary component on the C node with a following statement: SET CLUSTER posts GLOBAL 'pc.bootstrap' = 1  But someone must make sure that all the other nodes are really unreachable before doing that, otherwise split-brain happens and separate clusters get formed. ##### Case 6¶ All nodes crashed. In this case the grastate.dat file at cluster directory is not updated and does not contain a valid sequence number seqno. If this happened, someone should find the most advanced node and start the daemon on it with the –new-cluster-force command line key. All other nodes will start as usual as in case 3. ##### Case 7¶ Split-brain causes a cluster to get into non-primary state. For example, the cluster consists of even number of nodes (four), two couple of nodes being located in separate datacenters, and network failure interrupts the connection these datacenters. Split-brain happens as each group of nodes has exactly half of quorum. Both groups stop to handle write transactions as Galera replication model cares about data consistency and the cluster can not accept write transactions without quorum. But nodes in both groups try to re-connect to the nodes nodes from the other group to restore the cluster. If someone wants to restore the cluster without network got restored the same steps as in case 5 can be done but only at one group of nodes SET CLUSTER posts GLOBAL 'pc.bootstrap' = 1  After that, the group with the node we run this statement at can successfully handle write transactions again. However, we want to notice that if the statement gets issued at both groups this will end up with two separate clusters made, so the following network restoration will not make the groups to rejoin. ## Command line tools reference¶ As mentioned elsewhere, Manticore is not a single program called ‘sphinx’, but a collection of 4 separate programs which collectively form Manticore. This section covers these tools and how to use them. ### index_converter command reference¶ index_converter is a tool for converting indexes created with Sphinx/Manticore Search 2.x to Manticore Search 3.x index format. The tool can be used in several different ways: Convert one index at a time: $ index_converter --config /home/myuser/sphinx.conf --index indexname


Convert all indexes:

$index_converter --config /home/myuser/sphinx.conf --all  Convert indexes found in a folder: $ index_converter  --path /var/lib/manticoresearch/data --all


New version of the index is written by default in the same folder. Previous version files are saved with .old extension in their name. An exception is .spp (hitlists) file which is the only index component that didn’t have any change in the new format.

You can save the new index version to a different folder using –output-dir option

$index_converter --config /home/myuser/sphinx.conf --all --output-dir /new/path  A special case is for indexes containing kill-lists. As the behaviour of how kill-lists works has changed (see killlist_target), the delta index should know which are the target indexes for applying the kill-lists There are 3 ways to have a converted index ready for setting targeted indexes for applying kill-lists: • Use –killlist-target when converting an index $ index_converter --config /home/myuser/sphinx.conf --index deltaindex --killlist-target mainindex:kl

• Add killlist_target in the configuration before doing the conversion
• use ALTER … KILLLIST_TARGET command after conversion

A complete list of index_converter options:

• --config <file> (-c <file> for short) tells index_converter to use the given file as its configuration. Normally, it will look for sphinx.conf in the installation directory (e.g. /usr/local/sphinx/etc/sphinx.conf if installed into /usr/local/sphinx), followed by the current directory you are in when calling index_converter from the shell.
• --index specifies which index should be converted
• --path - instead of using a config file, a path containing index(es) can be used
• --strip-path - strips path from filenames referenced by index: stopwords, exceptions and wordforms
• --large-docid - allows to convert documents with ids larger than 2^63 and display a warning, otherwise it will just exit on the large id with an error. This option was added as in Manticore 3.x doc ids are signed bigint, while previously they were unsigned
• --output-dir <dir> - writes the new files in a chosen folder rather than the same location as with the existing index files. When this option set, existing index files will remain untouched at their location.
• --all - converts all indexes from the config
• --killlist-target <targets> - sets the target indexes for which kill-lists will be applied. This option should be used only in conjunction with --index option

### indexer command reference¶

indexer is the first of the two principal tools as part of Manticore. Invoked from either the command line directly, or as part of a larger script, indexer is solely responsible for gathering the data that will be searchable.

The calling syntax for indexer is as follows:

indexer [OPTIONS] [indexname1 [indexname2 [...]]]


Essentially you would list the different possible indexes (that you would later make available to search) in sphinx.conf, so when calling indexer, as a minimum you need to be telling it what index (or indexes) you want to index.

If sphinx.conf contained details on 2 indexes, mybigindex and mysmallindex, you could do the following:

$indexer mybigindex$ indexer mysmallindex mybigindex


As part of the configuration file, sphinx.conf, you specify one or more indexes for your data. You might call indexer to reindex one of them, ad-hoc, or you can tell it to process all indexes - you are not limited to calling just one, or all at once, you can always pick some combination of the available indexes.

Wildcarding on index names is also supported. The following wildcard tokens can be used:

• ? matches any single character
• * matches any count of any characters
• % matches none or any single character
$indexer indexpart*main --rotate  The exit codes are as follows: • 0, everything went ok • 1, there was a problem while indexing (and if –rotate was specified, it was skipped) • 2, indexing went ok, but –rotate attempt failed The majority of the options for indexer are given in the configuration file, however there are some options you might need to specify on the command line as well, as they can affect how the indexing operation is performed. These options are: • --config <file> (-c <file> for short) tells indexer to use the given file as its configuration. Normally, it will look for sphinx.conf in the installation directory (e.g. /usr/local/sphinx/etc/sphinx.conf if installed into /usr/local/sphinx), followed by the current directory you are in when calling indexer from the shell. This is most of use in shared environments where the binary files are installed somewhere like /usr/local/sphinx/ but you want to provide users with the ability to make their own custom Manticore set-ups, or if you want to run multiple instances on a single server. In cases like those you could allow them to create their own sphinx.conf files and pass them to indexer with this option. For example: $ indexer --config /home/myuser/sphinx.conf myindex

• --all tells indexer to update every index listed in sphinx.conf, instead of listing individual indexes. This would be useful in small configurations, or cron-type or maintenance jobs where the entire index set will get rebuilt each day, or week, or whatever period is best. Example usage:

$indexer --config /home/myuser/sphinx.conf --all  • --rotate is used for rotating indexes. Unless you have the situation where you can take the search function offline without troubling users, you will almost certainly need to keep search running whilst indexing new documents. --rotate creates a second index, parallel to the first (in the same place, simply including .new in the filenames). Once complete, indexer notifies searchd via sending the SIGHUP signal, and searchd will attempt to rename the indexes (renaming the existing ones to include .old and renaming the .new to replace them), and then start serving from the newer files. Depending on the setting of seamless_rotate, there may be a slight delay in being able to search the newer indexes. In case multiple indexes are rotated at once which are chained by killlist_target relations, rotation will start with the indexes that are not targets and finish with the ones at the end of target chain. Example usage: $ indexer --rotate --all

• --quiet tells indexer not to output anything, unless there is an error. Again, most used for cron-type, or other script jobs where the output is irrelevant or unnecessary, except in the event of some kind of error. Example usage:

$indexer --rotate --all --quiet  • --noprogress does not display progress details as they occur; instead, the final status details (such as documents indexed, speed of indexing and so on are only reported at completion of indexing. In instances where the script is not being run on a console (or ‘tty’), this will be on by default. Example usage: $ indexer --rotate --all --noprogress

• --buildstops <outputfile.text> <N> reviews the index source, as if it were indexing the data, and produces a list of the terms that are being indexed. In other words, it produces a list of all the searchable terms that are becoming part of the index. Note; it does not update the index in question, it simply processes the data ‘as if’ it were indexing, including running queries defined with sql_query_pre or sql_query_post. outputfile.txt will contain the list of words, one per line, sorted by frequency with most frequent first, and N specifies the maximum number of words that will be listed; if sufficiently large to encompass every word in the index, only that many words will be returned. Such a dictionary list could be used for client application features around “Did you mean…” functionality, usually in conjunction with --buildfreqs, below. Example:

$indexer myindex --buildstops word_freq.txt 1000  This would produce a document in the current directory, word_freq.txt with the 1,000 most common words in ‘myindex’, ordered by most common first. Note that the file will pertain to the last index indexed when specified with multiple indexes or --all (i.e. the last one listed in the configuration file) • --buildfreqs works with --buildstops (and is ignored if --buildstops is not specified). As --buildstops provides the list of words used within the index, --buildfreqs adds the quantity present in the index, which would be useful in establishing whether certain words should be considered stopwords if they are too prevalent. It will also help with developing “Did you mean…” features where you can how much more common a given word compared to another, similar one. Example: $ indexer myindex --buildstops word_freq.txt 1000 --buildfreqs


This would produce the word_freq.txt as above, however after each word would be the number of times it occurred in the index in question.

• --merge <dst-index> <src-index> is used for physically merging indexes together, for example if you have a main+delta scheme, where the main index rarely changes, but the delta index is rebuilt frequently, and --merge would be used to combine the two. The operation moves from right to left - the contents of src-index get examined and physically combined with the contents of dst-index and the result is left in dst-index. In pseudo-code, it might be expressed as: dst-index += src-index An example:

$indexer --merge main delta --rotate  In the above example, where the main is the master, rarely modified index, and delta is the less frequently modified one, you might use the above to call indexer to combine the contents of the delta into the main index and rotate the indexes. • --merge-dst-range <attr> <min> <max> runs the filter range given upon merging. Specifically, as the merge is applied to the destination index (as part of --merge, and is ignored if --merge is not specified), indexer will also filter the documents ending up in the destination index, and only documents will pass through the filter given will end up in the final index. This could be used for example, in an index where there is a ‘deleted’ attribute, where 0 means ‘not deleted’. Such an index could be merged with: $ indexer --merge main delta --merge-dst-range deleted 0 0


Any documents marked as deleted (value 1) would be removed from the newly-merged destination index. It can be added several times to the command line, to add successive filters to the merge, all of which must be met in order for a document to become part of the final index.

• --merge-killlists (and its shorter alias --merge-klists) changes the way kill lists are processed when merging indexes. By default, both kill lists get discarded after a merge. That supports the most typical main+delta merge scenario. With this option enabled, however, kill lists from both indexes get concatenated and stored into the destination index. Note that a source (delta) index kill list will be used to suppress rows from a destination (main) index at all times.

• --keep-attrs allows to reuse existing attributes on reindexing. Whenever the index is rebuilt, each new document id is checked for presence in the “old” index, and if it already exists, its attributes are transferred to the “new” index; if not found, attributes from the new index are used. If the user has updated attributes in the index, but not in the actual source used for the index, all updates will be lost when reindexing; using –keep-attrs enables saving the updated attribute values from the previous index. It is possible to specify a path for index files to used instead of reference path from config:

indexer myindex --keep-attrs=/path/to/index/files

• --keep-attrs-names=<attributes list> allows to specify attributes to reuse from existing index on reindexing. By default all attributes from existed index reused at new “index”

indexer myindex --keep-attrs=/path/to/index/files --keep-attrs-names=update,state

• --dump-rows <FILE> dumps rows fetched by SQL source(s) into the specified file, in a MySQL compatible syntax. Resulting dumps are the exact representation of data as received by indexer and help to repeat indexing-time issues.

• --verbose [debug|debugv|debugvv] guarantees that every row that caused problems indexing (duplicate, zero, or missing document ID; or file field IO issues; etc) will be reported. By default, this option is off, and problem summaries may be reported instead. Also you can use one of the optional parameters (debug, debugv, or debugvv) and it will switch on debug output from different parts of indexing process. Thay are similar to searchd’s parameters –logdebug, –logdebugv, –logdebugvv, but cause output to stdout instead of logging.

• --sighup-each is useful when you are rebuilding many big indexes, and want each one rotated into searchd as soon as possible. With --sighup-each, indexer will send a SIGHUP signal to searchd after successfully completing the work on each index. (The default behavior is to send a single SIGHUP after all the indexes were built.)

• --nohup is useful when you want to check your index with indextool before actually rotating it. indexer won’t send SIGHUP if this option is on.

• --print-queries prints out SQL queries that indexer sends to the database, along with SQL connection and disconnection events. That is useful to diagnose and fix problems with SQL sources.

• --help (-h for short) lists all of the parameters that can be called in your particular build of indexer.

• -v show version information of your particular build of indexer.

### indextool command reference¶

indextool is one of the helper tools within the Manticore package. It is used to dump miscellaneous debug information about the physical index. (Additional functionality such as index verification is planned in the future, hence the indextool name rather than just indexdump.) Its general usage is:

indextool <command> [options]


Options apply to all commands:

• --config <file> (-c <file> for short) overrides the built-in config file names.
• --quiet (-q for short) keep indextool quiet - it will not output banner, etc.
• --help (-h for short) lists all of the parameters that can be called in your particular build of indextool.
• -v show version information of your particular build of indextool.

The commands are as follows:

• --checkconfig just loads and verifies the config file to check if it’s valid, without syntax errors.
• --buildidf DICTFILE1 [DICTFILE2 ...] --out IDFILE build IDF file from one or several dictionary dumps. Additional parameter -skip-uniq will skip unique (df=1) words.
• --build-infixes INDEXNAME build infixes for an existing dict=keywords index (upgrades .sph, .spi in place). You can use this option for legacy index files that already use dict=keywords, but now need to support infix searching too; updating the index files with indextool may prove easier or faster than regenerating them from scratch with indexer.
• --dumpheader FILENAME.sph quickly dumps the provided index header file without touching any other index files or even the configuration file. The report provides a breakdown of all the index settings, in particular the entire attribute and field list.
• --dumpconfig FILENAME.sph dumps the index definition from the given index header file in (almost) compliant sphinx.conf file format.
• --dumpheader INDEXNAME dumps index header by index name with looking up the header path in the configuration file.
• --dumpdict INDEXNAME dumps dictionary. Additional -stats switch will dump to dictionary the total number of documents. It is required for dictionary files that are used for creation of IDF files.
• --dumpdocids INDEXNAME dumps document IDs by index name.
• --dumphitlist INDEXNAME KEYWORD dumps all the hits (occurrences) of a given keyword in a given index, with keyword specified as text.
• --dumphitlist INDEXNAME --wordid ID dumps all the hits (occurrences) of a given keyword in a given index, with keyword specified as internal numeric ID.
• --fold INDEXNAME OPTFILE This options is useful too see how actually tokenizer proceeds input. You can feed indextool with text from file if specified or from stdin otherwise. The output will contain spaces instead of separators (accordingly to your charset_table settings) and lowercased letters in words.
• --html_strip INDEXNAME filters stdin using HTML stripper settings for a given index, and prints the filtering results to stdout. Note that the settings will be taken from sphinx.conf, and not the index header.
• --mergeidf NODE1.idf [NODE2.idf ...] --out GLOBAL.idf merge several .idf files into a single one. Additional parameter -skip-uniq will skip unique (df=1) words.
• --morph INDEXNAME applies morphology to the given stdin and prints the result to stdout.
• --check INDEXNAME checks the index data files for consistency errors that might be introduced either by bugs in indexer and/or hardware faults. --check also works on RT indexes, RAM and disk chunks.
• --strip-path strips the path names from all the file names referenced from the index (stopwords, wordforms, exceptions, etc). This is useful for checking indexes built on another machine with possibly different path layouts.
• --rotate works only with --check and defines whether to check index waiting for rotation, i.e. with .new extension. This is useful when you want to check your index before actually using it.
• --apply-killlists loads and applies kill-lists for all indexes listed in the config file. Changes are saved in .SPM files. Kill-list files (.SPK) are deleted. This can be useful if you want to move applying indexes from daemon startup to indexing stage.

### searchd command reference¶

searchd is the second of the two principle tools as part of Manticore. searchd is the part of the system which actually handles searches; it functions as a server and is responsible for receiving queries, processing them and returning a dataset back to the different APIs for client applications.

Unlike indexer, searchd is not designed to be run either from a regular script or command-line calling, but instead either as a daemon to be called from init.d (on Unix/Linux type systems) or to be called as a service (on Windows-type systems), so not all of the command line options will always apply, and so will be build-dependent.

Calling searchd is simply a case of:

$searchd [OPTIONS]  The options available to searchd on all builds are: • --help (-h for short) lists all of the parameters that can be called in your particular build of searchd. • -v show version information of your particular build of searchd. • --config <file> (-c <file> for short) tells searchd to use the given file as its configuration, just as with indexer above. • --stop is used to asynchronously stop searchd, using the details of the PID file as specified in the sphinx.conf file, so you may also need to confirm to searchd which configuration file to use with the --config option. NB, calling --stop will also make sure any changes applied to the indexes with :ref:UpdateAttributes() <update_attributes> will be applied to the index files themselves. Example: $ searchd --config /home/myuser/sphinx.conf --stop

• --stopwait is used to synchronously stop searchd. --stop essentially tells the running instance to exit (by sending it a SIGTERM) and then immediately returns. --stopwait will also attempt to wait until the running searchd instance actually finishes the shutdown (eg. saves all the pending attribute changes) and exits. Example:

$searchd --config /home/myuser/sphinx.conf --stopwait  Possible exit codes are as follows: • 0 on success; • 1 if connection to running searchd daemon failed; • 2 if daemon reported an error during shutdown; • 3 if daemon crashed during shutdown. • --status command is used to query running searchd instance status, using the connection details from the (optionally) provided configuration file. It will try to connect to the running instance using the first configured UNIX socket or TCP port. On success, it will query for a number of status and performance counter values and print them. You can use Status() API call to access the very same counters from your application. Examples: $ searchd --status
$searchd --config /home/myuser/sphinx.conf --status  • --pidfile is used to explicitly force using a PID file (where the searchd process number is stored) despite any other debugging options that say otherwise (for instance, --console). This is a debugging option. $ searchd --console --pidfile

• --console is used to force searchd into console mode; typically it will be running as a conventional server application, and will aim to dump information into the log files (as specified in sphinx.conf). Sometimes though, when debugging issues in the configuration or the daemon itself, or trying to diagnose hard-to-track-down problems, it may be easier to force it to dump information directly to the console/command line from which it is being called. Running in console mode also means that the process will not be forked (so searches are done in sequence) and logs will not be written to. (It should be noted that console mode is not the intended method for running searchd.) You can invoke it as such:

$searchd --config /home/myuser/sphinx.conf --console  • --logdebug, --logreplication, --logdebugv, and --logdebugvv options enable additional debug output in the daemon log. They differ by the logging verboseness level. These are debugging options, they pollute the log a lot, and thus they should not be normally enabled. (The normal use case for these is to enable them temporarily on request, to assist with some particularly complicated debugging session.) • --iostats is used in conjunction with the logging options (the query_log will need to have been activated in sphinx.conf) to provide more detailed information on a per-query basis as to the input/output operations carried out in the course of that query, with a slight performance hit and of course bigger logs. Note The IO statistics don’t include information about IO operations for attributes, as these are loaded with mmap(). Further details are available under the query log format section. You might start searchd thus: $ searchd --config /home/myuser/sphinx.conf --iostats

• --cpustats is used to provide actual CPU time report (in addition to wall time) in both query log file (for every given query) and status report (aggregated). It depends on clock_gettime() system call or fall back to less precise call on certain systems. You might start searchd thus:

$searchd --config /home/myuser/sphinx.conf --cpustats  • --port portnumber (-p for short) is used to specify the port that searchd should listen on, usually for debugging purposes. This will usually default to 9312, but sometimes you need to run it on a different port. Specifying it on the command line will override anything specified in the configuration file. The valid range is 0 to 65535, but ports numbered 1024 and below usually require a privileged account in order to run. An example of usage: $ searchd --port 9313

• --listen ( address ":" port | port | path ) [ ":" protocol ] (or -l for short) Works as --port, but allow you to specify not only the port, but full path, as IP address and port, or Unix-domain socket path, that searchd will listen on. Otherwords, you can specify either an IP address (or hostname) and port number, or just a port number, or Unix socket path. If you specify port number but not the address, searchd will listen on all network interfaces. Unix path is identified by a leading slash. As the last param you can also specify a protocol handler (listener) to be used for connections on this socket. Supported protocol values are ‘sphinx’ and ‘mysql41’ (MySQL protocol used since 4.1 upto at least 5.1).

• --force-preread forbids the daemon to serve any incoming connection until prereading of index files completes. By default, at startup the daemon accepts connections while index files are lazy loaded into memory.

• --index <index> (or -i <index> for short) forces this instance of searchd only to serve the specified index. Like --port, above, this is usually for debugging purposes; more long-term changes would generally be applied to the configuration file itself. Example usage:

$searchd --index myindex  • --strip-path strips the path names from all the file names referenced from the index (stopwords, wordforms, exceptions, etc). This is useful for picking up indexes built on another machine with possibly different path layouts. • --replay-flags=<OPTIONS> switch can be used to specify a list of extra binary log replay options. The supported options are: • accept-desc-timestamp, ignore descending transaction timestamps and replay such transactions anyway (the default behavior is to exit with an error). • ignore-open-errors, ignore missing binlog files (the default behavior is to exit with an error). Example: $ searchd --replay-flags=accept-desc-timestamp

• --coredump is used to enable save of core file or minidump of daemon on crash. Disabled by default to speed up of daemon restart on crash. This is useful for debugging purposes.

$searchd --config /home/myuser/sphinx.conf --coredump  • --new-cluster bootstraps a replication cluster and makes the daemon a reference node with cluster restart protection • --new-cluster-force bootstraps a replication cluster and makes the daemon a reference node bypassing cluster restart protection There are some options for searchd that are specific to Windows platforms, concerning handling as a service, and are only available in Windows binaries. Note that in Windows searchd will default to --console mode, unless you install it as a service. • --install installs searchd as a service into the Microsoft Management Console (Control Panel / Administrative Tools / Services). Any other parameters specified on the command line, where --install is specified will also become part of the command line on future starts of the service. For example, as a part of calling searchd, you will likely also need to specify the configuration file with --config, and you would do that as well as specifying --install. Once called, the usual start/stop facilities will become available via the management console, so any methods you could use for starting, stopping and restarting services would also apply to searchd. Example: C:\WINDOWS\system32> C:\Manticore\bin\searchd.exe --install --config C:\Manticore\sphinx.conf  If you want to have the I/O stats every time you start searchd, you need to specify its option on the same line as the --install command thus: C:\WINDOWS\system32> C:\Manticore\bin\searchd.exe --install --config C:\Manticore\sphinx.conf --iostats  • --delete removes the service from the Microsoft Management Console and other places where services are registered, after previously installed with --install. Note, this does not uninstall the software or delete the indexes. It means the service will not be called from the services systems, and will not be started on the machine’s next start. If currently running as a service, the current instance will not be terminated (until the next reboot, or until searchd is called with --stop). If the service was installed with a custom name (with --servicename), the same name will need to be specified with --servicename when calling to uninstall. Example: C:\WINDOWS\system32> C:\Manticore\bin\searchd.exe --delete  • --servicename <name> applies the given name to searchd when installing or deleting the service, as would appear in the Management Console; this will default to searchd, but if being deployed on servers where multiple administrators may log into the system, or a system with multiple searchd instances, a more descriptive name may be applicable. Note that unless combined with --install or --delete, this option does not do anything. Example: C:\WINDOWS\system32> C:\Manticore\bin\searchd.exe --install --config C:\Manticore\sphinx.conf --servicename ManticoreSearch  • --ntservice is the option that is passed by the Management Console to searchd to invoke it as a service on Windows platforms. It would not normally be necessary to call this directly; this would normally be called by Windows when the service would be started, although if you wanted to call this as a regular service from the command-line (as the complement to --console) you could do so in theory. • --safetrace forces searchd to only use system backtrace() call in crash reports. In certain (rare) scenarios, this might be a “safer” way to get that report. This is a debugging option. • --nodetach switch (Linux only) tells searchd not to detach into background. This will also cause log entry to be printed out to console. Query processing operates as usual. This is a debugging option. Last but not least, as every other daemon, searchd supports a number of signals. • SIGTERM • Initiates a clean shutdown. New queries will not be handled; but queries that are already started will not be forcibly interrupted. • SIGHUP • Initiates index rotation. Depending on the value of seamless_rotate setting, new queries might be shortly stalled; clients will receive temporary errors. • SIGUSR1 • Forces reopen of searchd log and query log files, letting you implement log file rotation. ### spelldump command reference¶ spelldump is one of the helper tools within the Manticore package. It is used to extract the contents of a dictionary file that uses ispell or MySpell format, which can help build word lists for wordforms - all of the possible forms are pre-built for you. Its general usage is: spelldump [options] <dictionary> <affix> [result] [locale-name]  The two main parameters are the dictionary’s main file and its affix file; usually these are named as [language-prefix].dict and [language-prefix].aff and will be available with most common Linux distributions, as well as various places online. [result] specifies where the dictionary data should be output to, and [locale-name] additionally specifies the locale details you wish to use. There is an additional option, -c [file], which specifies a file for case conversion details. Examples of its usage are: spelldump en.dict en.aff spelldump ru.dict ru.aff ru.txt ru_RU.CP1251 spelldump ru.dict ru.aff ru.txt .1251  The results file will contain a list of all the words in the dictionary in alphabetical order, output in the format of a wordforms file, which you can use to customize for your specific circumstances. An example of the result file: zone > zone zoned > zoned zoning > zoning  ### wordbreaker command reference¶ wordbreaker is one of the helper tools within the Manticore package. It is used to split compound words, as usual in URLs, into its component words. For example, this tool can split “lordoftherings” into its four component words, or “http://manofsteel.warnerbros.com” into “man of steel warner bros”. This helps searching, without requiring prefixes or infixes: searching for “sphinx” wouldn’t match “sphinxsearch” but if you break the compound word and index the separate components, you’ll get a match without the costs of prefix and infix larger index files. Examples of its usage are: echo manofsteel | bin/wordbreaker -dict dict.txt split man of steel  The input stream will be separated in words using the -dict dictionary file. In no dictionary specified, wordbreaker looks in the working folder for a wordbreaker-dict.txt file. (The dictionary should match the language of the compound word.) The split command breaks words from the standard input, and outputs the result in the standard output. There are also test and bench commands that let you test the splitting quality and benchmark the splitting functionality. Wordbreaker Wordbreaker needs a dictionary to recognize individual substrings within a string. To differentiate between different guesses, it uses the relative frequency of each word in the dictionary: higher frequency means higher split probability. You can generate such a file using the indexer tool, as in indexer --buildstops dict.txt 100000 --buildfreqs myindex -c /path/to/sphinx.conf  which will write the 100,000 most frequent words, along with their counts, from myindex into dict.txt. The output file is a text file, so you can edit it by hand, if need be, to add or remove words. ## SphinxQL reference¶ SphinxQL is our SQL dialect that exposes all of the search daemon functionality using a standard SQL syntax with a few Manticore-specific extensions. Everything available via the SphinxAPI is also available via SphinxQL but not vice versa; for instance, writes into RT indexes are only available via SphinxQL. This chapter documents supported SphinxQL statements syntax. ### ALTER syntax¶ ALTER TABLE index {ADD|DROP} COLUMN column_name [{INTEGER|INT|BIGINT|FLOAT|BOOL|MULTI|MULTI64|JSON|STRING}]  It supports adding one attribute at a time for both plain and RT indexes. The int, bigint, float, bool, multi-valued, multi-valued 64bit, json and string attribute types are supported. You can add json and string attributes, but you cannot modify their values. Implementation details. The querying of an index is impossible (because of a write lock) while adding a column. This may change in the future. The newly created attribute values are set to 0. ALTER will not work for distributed indexes and indexes without any attributes. DROP COLUMN will fail if an index has only one attribute. ALTER RTINDEX index RECONFIGURE  ALTER can also reconfigure an existing RT index, so that new tokenization, morphology, and other text processing settings from sphinx.conf take effect on the newly INSERT-ed rows, while retaining the existing rows as they were. Internally, it forcibly saves the current RAM chunk as a new disk chunk, and adjusts the index header, so that the new rows are tokenized using the new rules. Note that as the queries are currently parsed separately for every disk chunk, this might result in warnings regarding the keyword sets mismatch. mysql> desc plain; +------------+-----------+ | Field | Type | +------------+-----------+ | id | bigint | | text | field | | group_id | uint | | date_added | timestamp | +------------+-----------+ 4 rows in set (0.01 sec) mysql> alter table plain add column test integer; Query OK, 0 rows affected (0.04 sec) mysql> desc plain; +------------+-----------+ | Field | Type | +------------+-----------+ | id | bigint | | text | field | | group_id | uint | | date_added | timestamp | | test | uint | +------------+-----------+ 5 rows in set (0.00 sec) mysql> alter table plain drop column group_id; Query OK, 0 rows affected (0.01 sec) mysql> desc plain; +------------+-----------+ | Field | Type | +------------+-----------+ | id | bigint | | text | field | | date_added | timestamp | | test | uint | +------------+-----------+ 4 rows in set (0.00 sec)  ALTER can be used to change killlist_target settings of an index: ALTER TABLE delta KILLLIST_TARGET='main'  Kill-list is applied to indexes listed in killlist_target immediately. ### ATTACH INDEX syntax¶ ATTACH INDEX diskindex TO RTINDEX rtindex [WITH TRUNCATE]  ATTACH INDEX statement lets you move data from a regular disk index to a RT index. After a successful ATTACH, the data originally stored in the source disk index becomes a part of the target RT index, and the source disk index becomes unavailable (until the next rebuild). ATTACH does not result in any index data changes. Basically, it just renames the files (making the source index a new disk chunk of the target RT index), and updates the metadata. So it is a generally quick operation which might (frequently) complete as fast as under a second. Note that when an index is attached to an empty RT index, the fields, attributes, and text processing settings (tokenizer, wordforms, etc) from the source index are copied over and take effect. The respective parts of the RT index definition from the configuration file will be ignored. When TRUNCATE option is used RT index got truncated prior to attaching source disk index. This allows to make operation atomic or make sure that attached source disk index will be only data at target RT index. ATTACH INDEX comes with a number of restrictions. Most notably, the target RT index is currently required to be either empty or have same setting as source disk index. In case source disk index got attached to non empty RT index, RT index data collected so far got stored as regular disk chunk then source disk index become newest disk chunk and documents with same ID from previous disk chunks got killed. The complete list is as follows. • Target RT index needs to be either empty or have same settings (See TRUNCATE RTINDEX syntax) • Source disk index needs to have boundary_step=0, stopword_step=1. mysql> DESC rt; +-----------+---------+ | Field | Type | +-----------+---------+ | id | integer | | testfield | field | | testattr | uint | +-----------+---------+ 3 rows in set (0.00 sec) mysql> SELECT * FROM rt; Empty set (0.00 sec) mysql> SELECT * FROM disk WHERE MATCH('test'); +------+--------+----------+------------+ | id | weight | group_id | date_added | +------+--------+----------+------------+ | 1 | 1304 | 1 | 1313643256 | | 2 | 1304 | 1 | 1313643256 | | 3 | 1304 | 1 | 1313643256 | | 4 | 1304 | 1 | 1313643256 | +------+--------+----------+------------+ 4 rows in set (0.00 sec) mysql> ATTACH INDEX disk TO RTINDEX rt; Query OK, 0 rows affected (0.00 sec) mysql> DESC rt; +------------+-----------+ | Field | Type | +------------+-----------+ | id | integer | | title | field | | content | field | | group_id | uint | | date_added | timestamp | +------------+-----------+ 5 rows in set (0.00 sec) mysql> SELECT * FROM rt WHERE MATCH('test'); +------+--------+----------+------------+ | id | weight | group_id | date_added | +------+--------+----------+------------+ | 1 | 1304 | 1 | 1313643256 | | 2 | 1304 | 1 | 1313643256 | | 3 | 1304 | 1 | 1313643256 | | 4 | 1304 | 1 | 1313643256 | +------+--------+----------+------------+ 4 rows in set (0.00 sec) mysql> SELECT * FROM disk WHERE MATCH('test'); ERROR 1064 (42000): no enabled local indexes to search  ### BEGIN, COMMIT, and ROLLBACK syntax¶ START TRANSACTION | BEGIN COMMIT ROLLBACK SET AUTOCOMMIT = {0 | 1}  BEGIN statement (or its START TRANSACTION alias) forcibly commits pending transaction, if any, and begins a new one. COMMIT statement commits the current transaction, making all its changes permanent. ROLLBACK statement rolls back the current transaction, canceling all its changes. SET AUTOCOMMIT controls the autocommit mode in the active session. AUTOCOMMIT is set to 1 by default, meaning that every statement that performs any changes on any index is implicitly wrapped in BEGIN and COMMIT. Transactions are limited to a single RT or percolate index, and also limited in size. They are atomic, consistent, overly isolated, and durable. Overly isolated means that the changes are not only invisible to the concurrent transactions but even to the current session itself. ### BEGIN syntax¶ START TRANSACTION | BEGIN  BEGIN syntax is discussed in detail in BEGIN, COMMIT, and ROLLBACK syntax. ### CALL KEYWORDS syntax¶ CALL KEYWORDS(text, index [, options])  CALL KEYWORDS statement splits text into particular keywords. It returns tokenized and normalized forms of the keywords, and, optionally, keyword statistics. It also returns the position of each keyword in the query and all forms of tokenized keywords in the case that lemmatizers were used. text is the text to break down to keywords. index is the name of the index from which to take the text processing settings. options, is an optional boolean parameter that specifies whether to return document and hit occurrence statistics. options can also accept parameters for configuring folding depending on tokenization settings: • stats - show statistics of keywords, default is 0 • fold_wildcards - fold wildcards, default is 1 • fold_lemmas - fold morphological lemmas, default is 0 • fold_blended - fold blended words, default is 0 • expansion_limit - override expansion_limit defined in configuration, default is 0 (use value from configuration) call keywords( 'que*', 'myindex', 1 as fold_wildcards, 1 as fold_lemmas, 1 as fold_blended, 1 as expansion_limit, 1 as stats);  Default values to match previous CALL KEYWORDS output are: call keywords( 'que*', 'myindex', 1 as fold_wildcards, 0 as fold_lemmas, 0 as fold_blended, 0 as expansion_limit, 0 as stats);  ### CALL PQ syntax¶ CALL PQ(index, data[, opt_value AS opt_name[, ...]])  CALL PQ statement performs a prospective search. It returns stored queries from a percolateindex that match documents from provideddata. For more information, see Percolate Query section. data can be: • a document in plain text • a JSON object containing a document • a JSON array of JSON documents • a list of any of the above (JSON and plain text documents cannot be mixed) The JSON object can contain pairs of text field names and values as well as attribute names and values. Examples: CALL PQ ('index_name', 'single document', 0 AS docs_json); CALL PQ ('index_name', ('first document', 'second document'), 0 AS docs_json ); CALL PQ ('index_name', '{"title":"single document","content":"Add your content here","category":10,"timestamp":1513725448}'); CALL PQ ('index_name', ( '{"title":"first document","content":"Add your content here","category":10,"timestamp":1513725448}', '{"title":"second document","content":"Add more content here","category":20,"timestamp":1513758240}' ) ); CALL PQ ('pq_sina', '[ {"title":"first document","content":"Add your content here","category":10,"timestamp":1513725448}, {"title":"second document","content":"Add more content here","category":20,"timestamp":1513758240} ]')  A number of options can be set: • docs - 0 (disabled by default), provides numbers of matched documents (in accordance with the order in the CALL PQ or document ids if docs_id is used) • docs_id - none (disabled by default), defines document id alias name from the JSON object to consider that as a document id (makes sense only with docs=1) • docs_json - 1 (enabled by default), specifies if the data provides document(s) as a raw string or as a JSON object. Besides single objects you can also provide array of them, using json array syntax [{…},{…},…]. • mode - none (‘sparsed’ by default), specifies how to distribute provided docs among members of distributed indexes. Available values are ‘sparsed’ and ‘sharded’. The details are described below in Distributed PQ modes section. • query - 0 (disabled by default), if true returns all information of matched stored queries, otherwise it returns just the stored query IDs. • skip_bad_json - 0 (disabled by default), specifies what to do if json document is broken: either immediately stop with an error message or just skip it and continue to process the rest of the documents. • skip_empty - 0 (disabled by default), specifies how to behave if we met totally empty json, i.e. just null or empty line. By default it is accounted as ‘bad json’ and so, may be managed by skip_bad_json as any other kind of errors. Setting it to 1 will silently ignore such documents without any errors and even warnings. • shift - 0 by default, defines the number which will be added to document ids if no docs_id fields provided. Makes sense mainly to support Distributed PQ modes. • verbose - 0 (disabled by default), provides extended info in SHOW META The output of CALL PQ return the following columns: • id - the id of the stored query • documents - if docs_id is not set, it will return indexes of the documents as defined at input. If docs_id is set, the document indexes will be replaced with the values of the field defined by docs_id • query - the stored full text query • tags - the tags attached to the stored query • filters - the filters attached to the stored query Example: mysql> CALL PQ ('pq', ('{"title":"angry test", "gid":3 }', '{"title":"filter test doc2", "gid":13}'), 1 AS docs, 1 AS verbose, 1 AS query); +------+-----------+-------------+------+-------------------+ | id | documents | query | tags | filters | +------+-----------+-------------+------+-------------------+ | 1 | 2 | filter test | bla | gid>=10 | | 2 | 1 | angry | | gid>=10 OR gid<=3 | +------+-----------+-------------+------+-------------------+ 2 rows in set (0.00 sec)  CALL PQ can be followed by a SHOW META statement which provides additional meta-information about the executed prospective search. #### Distributed PQ modes¶ CALL PQ transparently works with both local percolate indexes (defined in config under type percolate), and distributed indexes consisting of local and remote percolate indexes or their combination. However, for more effective work you can organize your distributed indexes using two different approaches: 1. Sparsed. Batch of documents you pass in CALL PQ will be split into parts according to the number of agents, so each of the nodes will receive and process only a part of the documents from your request. To distinguish between the parts each agent will also receive param shift. 2. Sharded. The whole CALL PQ will be just broadcasted to all agents, without any initial documents split. Sparsed will be beneficial when your set of documents you send to call pq is quite big, but the set of queries stored in pq index is quite small. Assuming that all the hosts are mirrors Manticore will split your set of documents and distribute the chunks among the mirrors. Once the agents are done with the queries it will collect and merge all the results and return final query set as if it comes from one solid index. Let’s assume you have index pq_d2 which is defined in config as index pq_d2 { type = distributed agent = 127.0.0.1:6712:pq agent = 127.0.0.1:6712:pq1 }  Each of ‘pq’ and ‘pq1’ contains: mysql> SELECY * FROM pq; +------+-------------+------+-------------------+ | id | query | tags | filters | +------+-------------+------+-------------------+ | 1 | filter test | | gid>=10 | | 2 | angry | | gid>=10 OR gid<=3 | +------+-------------+------+-------------------+ 2 rows in set (0.01 sec)  And you fire CALL PQ to the distributed index with a couple of docs. It will return: mysql> CALL PQ ('pq_d2', ('{"title":"angry test", "gid":3 }', '{"title":"filter test doc2", "gid":13}'), 1 AS docs); +------+-----------+ | id | documents | +------+-----------+ | 1 | 2 | | 2 | 1 | +------+-----------+  In sparsed mode the head search deamon (the one to which you connect and invoke CALL PQ) will distribute the incoming batch of docs among the agents: ‘{“title”:”angry test”, “gid”:3 }’ will be sent to the first, and ‘{“title”:”filter test doc2”, “gid”:13}, 1 as shift’ to the second. So each of agents gets only half of all the documents. They then process the statements and return the results back to the head. If the documents don’t contain explicitly defined docs_id field, each agent in advance will add the value of shift to the calculated docid values. On return, the head daemon merges results and returns them to you. So you see the same result as if you invoked CALL PQ to a single local pq index, but actually the work was distributed and each node made half of that. Sharded mode is beneficial when you push relatively small set of documents, but the number of stored queries is huge. So in this case it is more appropriate to store just part of PQ rules on each node and then merge the results returned from the nodes that process one and the same set of documents against different sets of PQ rules. This mode has to be explicitly set since first of all it implies multiplication of network payload and secondly it expects different indexes in terms of PQ rules in each of the remote agents. The payload multiplication is absolutely useless if your remotes all have one and the same index (well, they will answer one and the same result, so why sending the whole set to _each_ of them?). Note that the query mode (sharded or sparsed) cannot be specified in the config. You have to choose the desired mode when creating and filling PQ indexes by analysing metrics. Some research may be required to make sure you benefit from either of the modes. Note that the syntax of HA mirrors in the config (when several hosts are assigned to one agent line, separated with | ) has nothing to do with the CALL PQ query mode. (so each agent always represents ONE host node of dpq despite of the number of HA mirrors specified for this agent). ### CALL QSUGGEST syntax¶ CALL QSUGGEST(word, index [,options])  CALL QSUGGEST statement enumerates for a giving word all suggestions from the dictionary. This statement works only on indexes with infixing enabled and dict=keywords. It returns the suggested keywords, Levenshtein distance between the suggested and original keyword and the docs statistic of the suggested keyword. If the first parameter is a bag of words, the function will return suggestions only for the last word, ignoring the rest. Several options are supported for customization: • limit - returned N top matches, default is 5 • max_edits - keep only dictionary words which Levenshtein distance is less or equal, default is 4 • result_stats - provide Levenshtein distance and document count of the found words, default is 1 (enabled) • delta_len - keep only dictionary words whose length difference is less, default is 3 • max_matches - number of matches to keep, default is 25 • reject - defaults to 4; rejected words are matches that are not better than those already in the match queue. They are put in a rejected queue that gets reset in case one actually can go in the match queue. This parameter defines the size of the rejected queue (as reject*max(max_matched,limit)). If the rejected queue is filled, the engine stops looking for potential matches. • result_line - alternate mode to display the data by returning all suggests, distances and docs each per one row, default is 0 • non_char - do not skip dictionary words with non alphabet symbols, default is 0 (skip such words) mysql> CALL QSUGGEST('automaticlly ','forum', 5 as limit, 4 as max_edits,1 as result_stats,3 as delta_len,0 as result_line,25 as max_matches,4 as reject ); +---------------+----------+------+ | suggest | distance | docs | +---------------+----------+------+ | automatically | 1 | 282 | | automaticly | 1 | 6 | | automaticaly | 1 | 3 | | automagically | 2 | 14 | | automtically | 2 | 1 | +---------------+----------+------+ 5 rows in set (0.00 sec)  ### CALL SNIPPETS syntax¶ CALL SNIPPETS(data, index, query[, opt_value AS opt_name[, ...]])  CALL SNIPPETS statement builds a snippet from provided data and query, using specified index settings. data is the source data to extract a snippet from. It could be a single string, or the list of the strings enclosed in curly brackets. index is the name of the index from which to take the text processing settings. query is the full-text query to build snippets for. Additional options are documented in BuildExcerpts. Usage example: CALL SNIPPETS('this is my document text', 'test1', 'hello world', 5 AS around, 200 AS limit); CALL SNIPPETS(('this is my document text','this is my another text'), 'test1', 'hello world', 5 AS around, 200 AS limit); CALL SNIPPETS(('data/doc1.txt','data/doc2.txt','/home/sphinx/doc3.txt'), 'test1', 'hello world', 5 AS around, 200 AS limit, 1 AS load_files);  ### CALL SUGGEST syntax¶ CALL SUGGEST(word, index [,options])  CALL SUGGEST statement works the same as CALL QSUGGEST, except that if a bag of words is present, the statement will return suggestions only for the first word, ignoring the rest. If the first paramenter is a word, the functionality of CALL SUGGEST and CALL QSUGGEST is the same. ### Comment syntax¶ SphinxQL supports C-style comment syntax. Everything from an opening /* sequence to a closing */ sequence is ignored. Comments can span multiple lines, can not nest, and should not get logged. MySQL specific /*! ... */ comments are also currently ignored. (As the comments support was rather added for better compatibility with mysqldump produced dumps, rather than improving general query interoperability between Manticore and MySQL.) SELECT /*! SQL_CALC_FOUND_ROWS */ col1 FROM table1 WHERE ...  ### CREATE FUNCTION syntax¶ CREATE FUNCTION udf_name RETURNS {INT | INTEGER | BIGINT | FLOAT | STRING} SONAME 'udf_lib_file'  CREATE FUNCTION statement installs a user-defined function (UDF) with the given name and type from the given library file. The library file must reside in a trusted plugin_dir directory. On success, the function is available for use in all subsequent queries that the server receives. Example: mysql> CREATE FUNCTION avgmva RETURNS INTEGER SONAME 'udfexample.dll'; Query OK, 0 rows affected (0.03 sec) mysql> SELECT *, AVGMVA(tag) AS q from test1; +------+--------+---------+-----------+ | id | weight | tag | q | +------+--------+---------+-----------+ | 1 | 1 | 1,3,5,7 | 4.000000 | | 2 | 1 | 2,4,6 | 4.000000 | | 3 | 1 | 15 | 15.000000 | | 4 | 1 | 7,40 | 23.500000 | +------+--------+---------+-----------+  ### CREATE PLUGIN syntax¶ CREATE PLUGIN plugin_name TYPE 'plugin_type' SONAME 'plugin_library'  Loads the given library (if it is not loaded yet) and loads the specified plugin from it. The known plugin types are: • ranker • index_token_filter • query_token_filter Refer to Plugins for more information regarding writing the plugins. mysql> CREATE PLUGIN myranker TYPE 'ranker' SONAME 'myplugins.so'; Query OK, 0 rows affected (0.00 sec)  ### DEBUG syntax¶ DEBUG [ subcommand ]  DEBUG statement is designed to call different internal or vip commands for dev/testing purposes. It is not intended for production automation, since the syntax of subcommand part may be freely changed in any build. Call DEBUG without params to show list of useful commands (in general) and subcommands (of DEBUG statement) available at the moment. However you can invoke DEBUG without params to know which subcommands of the statement are available in any particular case: mysql> debug; +----------------+---------------------+ | command | meaning | +----------------+---------------------+ | flush logs | emulate USR1 signal | | reload indexes | emulate HUP signal | +----------------+---------------------+ 2 rows in set (0,00 sec)  (these commands are already documented, but such short help just remind about them). If you connect via ‘VIP’ connection (see listen for details) the output might be a bit different: mysql> debug; +---------------------------+------------------------------+ | command | meaning | +---------------------------+------------------------------+ | debug shutdown <password> | emulate TERM signal | | debug token <password> | calculate token for password | | flush logs | emulate USR1 signal | | reload indexes | emulate HUP signal | +---------------------------+------------------------------+ 4 rows in set (0,00 sec)  Here you can see additional commands available only in the current context (namely, if you connected on a VIP port). Two additional subcommands available right now are token and shutdown. The first one just calculates a hash (SHA1) of the <password> (which, in turn, may be empty, or a word, or num/phrase enclosed in ‘-quotes) like: mysql> debug token hello; +-------------+------------------------------------------+ | command | result | +-------------+------------------------------------------+ | debug token | aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d | +-------------+------------------------------------------+ 1 row in set (0,00 sec)  Another debug subcommand shutdown will send a TERM signal to the daemon and so will make it shut down. Since it is quite dangerous (nobody wants accidentally stop a production service), it 1) needs a VIP connection, and 2) needs the password. For the chosen password you need to generate a token with debug token subcommand, and put it into ref:shutdown_token param of searchd section of the config file. If no such section exists, or if a hash of the provided password does not match with the token stored in the config, the subcommand will do nothing. Otherwise it will cause ‘clean’ shutdown of the daemon. ### DELETE syntax¶ DELETE FROM index WHERE where_condition  DELETE statement is only supported for RT indexes and for distributed which contains only RT indexes as agents It deletes existing rows (documents) from an existing index based on ID. index is the name of RT index from which the row should be deleted. where_condition has the same syntax as in the SELECT statement (see SELECT syntax for details). mysql> select * from rt; +------+------+-------------+------+ | id | gid | mva1 | mva2 | +------+------+-------------+------+ | 100 | 1000 | 100,201 | 100 | | 101 | 1001 | 101,202 | 101 | | 102 | 1002 | 102,203 | 102 | | 103 | 1003 | 103,204 | 103 | | 104 | 1004 | 104,204,205 | 104 | | 105 | 1005 | 105,206 | 105 | | 106 | 1006 | 106,207 | 106 | | 107 | 1007 | 107,208 | 107 | +------+------+-------------+------+ 8 rows in set (0.00 sec) mysql> delete from rt where match ('dumy') and mva1>206; Query OK, 2 rows affected (0.00 sec) mysql> select * from rt; +------+------+-------------+------+ | id | gid | mva1 | mva2 | +------+------+-------------+------+ | 100 | 1000 | 100,201 | 100 | | 101 | 1001 | 101,202 | 101 | | 102 | 1002 | 102,203 | 102 | | 103 | 1003 | 103,204 | 103 | | 104 | 1004 | 104,204,205 | 104 | | 105 | 1005 | 105,206 | 105 | +------+------+-------------+------+ 6 rows in set (0.00 sec) mysql> delete from rt where id in (100,104,105); Query OK, 3 rows affected (0.01 sec) mysql> select * from rt; +------+------+---------+------+ | id | gid | mva1 | mva2 | +------+------+---------+------+ | 101 | 1001 | 101,202 | 101 | | 102 | 1002 | 102,203 | 102 | | 103 | 1003 | 103,204 | 103 | +------+------+---------+------+ 3 rows in set (0.00 sec) mysql> delete from rt where mva1 in (102,204); Query OK, 2 rows affected (0.01 sec) mysql> select * from rt; +------+------+---------+------+ | id | gid | mva1 | mva2 | +------+------+---------+------+ | 101 | 1001 | 101,202 | 101 | +------+------+---------+------+ 1 row in set (0.00 sec)  ### DESCRIBE syntax¶ {DESC | DESCRIBE} index [ LIKE pattern ]  DESCRIBE statement lists index columns and their associated types. Columns are document ID, full-text fields, and attributes. The order matches that in which fields and attributes are expected by INSERT and REPLACE statements. Column types are field, integer, timestamp, ordinal, bool, float, bigint, string, and mva. ID column will be typed as bigint. Example: mysql> DESC rt; +---------+---------+ | Field | Type | +---------+---------+ | id | bigint | | title | field | | content | field | | gid | integer | +---------+---------+ 4 rows in set (0.00 sec)  An optional LIKE clause is supported. Refer to SHOW META syntax for its syntax details. ### Percolate index schemas¶ If you apply DESC statement to a percolate index it will show the outer schema which is used to view the stored queries. That schema is fixed and the same for all local pq indexes: mysql> DESC pq; +---------+--------+ | Field | Type | +---------+--------+ | id | bigint | | query | string | | tags | string | | filters | string | +---------+--------+ 4 rows in set (0.00 sec)  If you’re looking for an expected document schema use DESC <pq index name> table: mysql> DESC pq TABLE; +-------+--------+ | Field | Type | +-------+--------+ | id | bigint | | title | field | | gid | uint | +-------+--------+ 3 rows in set (0.00 sec)  Also desc pq table like ... is possible and works as expected. ### DROP FUNCTION syntax¶ DROP FUNCTION udf_name  DROP FUNCTION statement deinstalls a user-defined function (UDF) with the given name. On success, the function is no longer available for use in subsequent queries. Pending concurrent queries will not be affected and the library unload, if necessary, will be postponed until those queries complete. Example: mysql> DROP FUNCTION avgmva; Query OK, 0 rows affected (0.00 sec)  ### DROP PLUGIN syntax¶ DROP PLUGIN plugin_name TYPE 'plugin_type'  Markes the specified plugin for unloading. The unloading is not immediate, because the concurrent queries might be using it. However, after a DROP new queries will not be able to use it. Then, once all the currently executing queries using it are completed, the plugin will be unloaded. Once all the plugins from the given library are unloaded, the library is also automatically unloaded. mysql> DROP PLUGIN myranker TYPE 'ranker'; Query OK, 0 rows affected (0.00 sec)  ### FLUSH ATTRIBUTES syntax¶ FLUSH ATTRIBUTES  Flushes all in-memory attribute updates in all the active disk indexes to disk. Returns a tag that identifies the result on-disk state (basically, a number of actual disk attribute saves performed since the daemon startup). mysql> UPDATE testindex SET channel_id=1107025 WHERE id=1; Query OK, 1 row affected (0.04 sec) mysql> FLUSH ATTRIBUTES; +------+ | tag | +------+ | 1 | +------+ 1 row in set (0.19 sec)  ### FLUSH HOSTNAMES syntax¶ FLUSH HOSTNAMES  Renew IPs associates to agent host names. To always query the DNS for getting the host name IP, see hostname_lookup directive. mysql> FLUSH HOSTNAMES; Query OK, 5 rows affected (0.01 sec)  ### FLUSH LOGS syntax¶ FLUSH LOGS  Works same as system USR1 signal. Initiate reopen of searchd log and query log files, letting you implement log file rotation. Command is non-blocking (i.e., returns immediately). mysql> FLUSH LOGS; Query OK, 0 rows affected (0.01 sec)  ### FLUSH RAMCHUNK syntax¶ FLUSH RAMCHUNK rtindex  FLUSH RAMCHUNK forcibly creates a new disk chunk in a RT index. Normally, RT index would flush and convert the contents of the RAM chunk into a new disk chunk automatically, once the RAM chunk reaches the maximum allowed rt_mem_limit size. However, for debugging and testing it might be useful to forcibly create a new disk chunk, and FLUSH RAMCHUNK statement does exactly that. Note that using FLUSH RAMCHUNK increases RT index fragmentation. Most likely, you want to use FLUSH RTINDEX instead. We suggest that you abstain from using just this statement unless you’re absolutely sure what you’re doing. As the right way is to issue FLUSH RAMCHUNK with following OPTIMIZE command. Such combo allows to keep RT index fragmentation on minimum. mysql> FLUSH RAMCHUNK rt; Query OK, 0 rows affected (0.05 sec)  ### FLUSH RTINDEX syntax¶ FLUSH RTINDEX rtindex  FLUSH RTINDEX forcibly flushes RT index RAM chunk contents to disk. Backing up a RT index is as simple as copying over its data files, followed by the binary log. However, recovering from that backup means that all the transactions in the log since the last successful RAM chunk write would need to be replayed. Those writes normally happen either on a clean shutdown, or periodically with a (big enough!) interval between writes specified in rt_flush_period directive. So such a backup made at an arbitrary point in time just might end up with way too much binary log data to replay. FLUSH RTINDEX forcibly writes the RAM chunk contents to disk, and also causes the subsequent cleanup of (now-redundant) binary log files. Thus, recovering from a backup made just after FLUSH RTINDEX should be almost instant. mysql> FLUSH RTINDEX rt; Query OK, 0 rows affected (0.05 sec)  ### INSERT and REPLACE syntax¶ {INSERT | REPLACE} INTO index [(column, ...)] VALUES (value, ...) [, (...)]  INSERT statement is only supported for RT and percolate indexes. It inserts new rows (documents) into an existing index, with the provided column values. index is the name of RT or percolate index into which the new row(s) should be inserted. The optional column names list lets you only explicitly specify values for some of the columns present in the index. All the other columns will be filled with their default values (0 for scalar types, empty string for text types). Expressions are not currently supported in INSERT and values should be explicitly specified. #### RT index INSERT features¶ ID column is mandatory for RT indexes. Rows with duplicate IDs will not be overwritten by INSERT; use REPLACE to do that. REPLACE works exactly like INSERT, except that if an old row has the same ID as a new row, the old row is deleted before the new row is inserted. Multiple rows can be inserted using a single INSERT statement by providing several comma-separated, parentheses-enclosed lists of rows values. #### Percolate index INSERT features¶ For percolate indexes INSERT is used to store queries (aka PQ rules) and their meta (id, tags), so the schema is predefined and may include only the following columns: • id - numeric id of stored query (if omited, will be assigned automatically) • query - full-text query to store • filters - filters to store (without query will define the full query as full-scan) • tags - string with one or many comma-separated tags, which may be used to selectively show/delete saved queries. All other names for columns are not supported and will trigger an error. INSERT INTO pq (id, query, filters) VALUES ( 1, 'filter test', 'gid >= 10' ) INSERT INTO index_name (query) VALUES ( 'full text query terms' ); INSERT INTO index_name (query, tags, filters) VALUES ( 'full text query terms', 'tags', 'filters' );  In case of omitted schema INSERT expects one or two params, first is full-text query, and second (optional) is tags. id in this case will be generated automatically (maximum current id in the index + 1), filters will be empty. INSERT INTO index_name VALUES ( 'full text query terms', 'tags'); INSERT INTO index_name VALUES ( 'full text query terms');  Multiple rows can be inserted using a single INSERT statement by providing several comma-separated, parentheses-enclosed lists of rows values. Also, you can insert values only into local percolate index. Distributed percolate (i.e. distributed index built from percolate agents/locals) is not supported for INSERTs yet. ### List of SphinxQL reserved keywords¶ A complete alphabetical list of keywords that are currently reserved in SphinxQL syntax (and therefore can not be used as identifiers). AND, AS, AT, BY, DEBUG, DIV, FACET, FALSE, FROM, FORCE, IGNORE, IN, INDEXES, IS, JOIN, LIMIT, LOGS, MOD, NOT, NULL, OR, ORDER, REGEX, RELOAD, SELECT, SYSFILTERS, TRUE, USE, WAIT_TIMEOUT, KILLLIST_TARGET  ### Multi-statement queries¶ SphinxQL supports multi-statement queries, or batches. Possible inter-statement optimizations described in Multi-queries do apply to SphinxQL just as well. The batched queries should be separated by a semicolon. Your MySQL client library needs to support MySQL multi-query mechanism and multiple result set. For instance, mysqli interface in PHP and DBI/DBD libraries in Perl are known to work. Here’s a PHP sample showing how to utilize mysqli interface with Manticore. <?php$link = mysqli_connect ( "127.0.0.1", "root", "", "", 9306 );
if ( mysqli_connect_errno() )
die ( "connect failed: " . mysqli_connect_error() );

$batch = "SELECT * FROM test1 ORDER BY group_id ASC;";$batch .= "SELECT * FROM test1 ORDER BY group_id DESC";

if ( !mysqli_multi_query ( $link,$batch ) )
die ( "query failed" );

do
{
// fetch and print result set
if ( $result = mysqli_store_result($link) )
{
while ( $row = mysqli_fetch_row($result) )
printf ( "id=%s\n", $row[0] ); mysqli_free_result($result);
}

// print divider
if ( mysqli_more_results($link) ) printf ( "------\n" ); } while ( mysqli_next_result($link) );


Its output with the sample test1 index included with Manticore is as follows.

\$ php test_multi.php
id=1
id=2
id=3
id=4
------
id=3
id=4
id=1
id=2


The following statements can currently be used in a batch: SELECT, SHOW WARNINGS, SHOW STATUS, and SHOW META. Arbitrary sequence of these statements are allowed. The results sets returned should match those that would be returned if the batched queries were sent one by one.

### OPTIMIZE INDEX syntax¶

OPTIMIZE INDEX index_name


OPTIMIZE statement enqueues a RT index for optimization in a background thread.

If OPTION sync=1 is used, the command will wait until the optimization process is done (or if the connection timeout - but the optimization will continue to run).

Over time, RT indexes can grow fragmented into many disk chunks and/or tainted with deleted, but unpurged data, impacting search performance. When that happens, they can be optimized. Basically, the optimization pass merges together disk chunks pairs, purging off documents suppressed by K-list as it goes.

That is a lengthy and IO intensive process, so to limit the impact, all the actual merge work is executed serially in a special background thread, and the OPTIMIZE statement simply adds a job to its queue. Currently, there is no way to check the index or queue status (that might be added in the future to the SHOW INDEX STATUS and SHOW STATUS statements respectively). The optimization thread can be IO-throttled, you can control the maximum number of IOs per second and the maximum IO size with rt_merge_iops and rt_merge_maxiosize directives respectively. The optimization jobs queue is lost on daemon crash.

The RT index being optimized stays online and available for both searching and updates at (almost) all times during the optimization. It gets locked (very) briefly every time that a pair of disk chunks is merged successfully, to rename the old and the new files, and update the index header.

At the moment, OPTIMIZE needs to be issued manually, the indexes will not be optimized automatically. That might change in the future releases.

mysql> OPTIMIZE INDEX rt;
Query OK, 0 rows affected (0.00 sec)


RELOAD INDEX idx [ FROM '/path/to/index_files' ]


RELOAD INDEX allows you to rotate indexes using SphinxQL.

It has two modes of operation. First one (without specifying a path) makes Manticore daemon check for new index files in directory specified in path. New index files must have a idx.new.sp? names.

And if you additionally specify a path, daemon will look for index files in specified directory, move them to index path, rename from index_files.sp? to idx.new.sp? and rotate them.

mysql> RELOAD INDEX plain_index;
mysql> RELOAD INDEX plain_index FROM '/home/mighty/new_index_files';


RELOAD INDEXES


Works same as system HUP signal. Initiates index rotation. Depending on the value of seamless_rotate setting, new queries might be shortly stalled; clients will receive temporary errors. Command is non-blocking (i.e., returns immediately).

mysql> RELOAD INDEXES;
Query OK, 0 rows affected (0.01 sec)


RELOAD PLUGINS FROM SONAME 'plugin_library'


Reloads all plugins (UDFs, rankers, etc) from a given library. Reload is, in a sense, transactional: a successful reload guarantees that a) all the plugins were successfully updated with their new versions; b) the update was atomic, all the plugins were replaced at once. Atomicity means that queries using multiple functions from a reloaded library will never mix the old and new versions. The set of plugins is guaranteed to always be consistent during the RELOAD, it will be either all old, or all new.

Reload also is seamless, meaning that some version of a reloaded plugin will be available to concurrent queries at all times, and there will be no temporary disruptions. Note how this improves on using a pair of DROP and CREATE statements for reloading: with those, there is a tiny window between the DROP and the subsequent CREATE, during which the queries technically refer to an unknown plugin and will thus fail.

In case of any failure RELOAD PLUGINS does absolutely nothing, keeps the old plugins, and reports an error.

On Windows, either overwriting or deleting a DLL library currently in use seems to be an issue. However, you can still rename it, then put a new version under the old name, and RELOAD will then work. After a succesful reload you will also be able to delete the renamed old library, too.

mysql> RELOAD PLUGINS FROM SONAME 'udfexample.dll';
Query OK, 0 rows affected (0.00 sec)


### REPLACE syntax¶

{INSERT | REPLACE} INTO index [(column, ...)]
VALUES (value, ...)
[, (...)]


REPLACE syntax is identical to INSERT syntax and is described in INSERT and REPLACE syntax.

### ROLLBACK syntax¶

ROLLBACK


ROLLBACK syntax is discussed in detail in BEGIN, COMMIT, and ROLLBACK syntax.

### SELECT syntax¶

SELECT
select_expr [, select_expr ...]
FROM index [, index2 ...]
[WHERE where_condition]
[GROUP [N] BY {col_name | expr_alias} [, {col_name | expr_alias}]]
[WITHIN GROUP ORDER BY {col_name | expr_alias} {ASC | DESC}]
[HAVING having_condition]
[ORDER BY {col_name | expr_alias} {ASC | DESC} [, ...]]
[LIMIT [offset,] row_count]
[OPTION opt_name = opt_value [, ...]]
[FACET facet_options[ FACET facet_options][ ...]]


SELECT statement’s syntax is based upon regular SQL but adds several Manticore-specific extensions and has a few omissions (such as (currently) missing support for JOINs). Specifically,

#### Column list¶

Column list clause. Column names, arbitrary expressions, and star (‘*’) are all allowed (ie. SELECT id, group_id*123+456 AS expr1 FROM test1 will work). Unlike in regular SQL, all computed expressions must be aliased with a valid identifier. AS is optional.

#### EXIST()¶

EXIST ( “attr-name”, default-value ) replaces non-existent columns with default values. It returns either a value of an attribute specified by ‘attr-name’, or ‘default-value’ if that attribute does not exist. It does not support STRING or MVA attributes. This function is handy when you are searching through several indexes with different schemas.

SELECT *, EXIST('gid', 6) as cnd FROM i1, i2 WHERE cnd>5


#### SNIPPET()¶

This is a wrapper around the snippets functionality, similar to what is available via CALL SNIPPETS. The first two arguments are: the text to highlight, and a query. It’s possible to pass options to function. The intended use is as follows:

SELECT id, SNIPPET(myUdf(id), 'my.query', 'limit=100')
FROM myIndex WHERE MATCH('my.query')


where myUdf() would be a UDF that fetches a document by its ID from some external storage. This enables applications to fetch the entire result set directly from Manticore in one query, without having to separately fetch the documents in the application and then send them back to Manticore for highlighting.

SNIPPET() is a so-called “post limit” function, meaning that computing snippets is postponed not just until the entire final result set is ready, but even after the LIMIT clause is applied. For example, with a LIMIT 20,10 clause, SNIPPET() will be called at most 10 times.

Table functions is a mechanism of post-query result set processing. Table functions take an arbitrary result set as their input, and return a new, processed set as their output. The first argument should be the input result set, but a table function can optionally take and handle more arguments. Table functions can completely change the result set, including the schema. For now, only built in table functions are supported. UDFs are planned when the internal call interface is stabilized. Table functions work for both outer SELECT and nested SELECT.

#### REMOVE_REPEATS()¶

REMOVE_REPEATS ( result_set, column, offset, limit ) - removes repeated adjusted rows with the same ‘column’ value.

SELECT REMOVE_REPEATS((SELECT * FROM dist1), gid, 0, 10)


#### FROM¶

FROM clause should contain the list of indexes to search through. Unlike in regular SQL, comma means enumeration of full-text indexes as in Query() API call rather than JOIN. Index name should be according to the rules of a C identifier.

#### WHERE¶

This clause will map both to fulltext query and filters. Comparison operators (=, !=, <, >, <=, >=), IN, AND, OR, NOT, BETWEEN and REGEX are all supported and map directly to filters. MATCH(‘query’) is supported and maps to fulltext query. Query will be interpreted according to full-text query language rules. There must be at most one MATCH() in the clause. {col_name | expr_alias} [NOT] IN @uservar condition syntax is supported. (Refer to SET syntax for a description of global user variables.)

#### GROUP BY¶

Supports grouping by multiple columns or computed expressions:

SELECT *, group_id*1000+article_type AS gkey FROM example GROUP BY gkey
SELECT id FROM products GROUP BY region, price


Implicit grouping supported when using aggregate functions without specifiying a GROUP BY clause. Consider these two queries:

SELECT MAX(id), MIN(id), COUNT(*) FROM books
SELECT MAX(id), MIN(id), COUNT(*), 1 AS grp FROM books GROUP BY grp


Aggregate functions (AVG(), MIN(), MAX(), SUM()) in column list clause are supported. Arguments to aggregate functions can be either plain attributes or arbitrary expressions. COUNT(*), COUNT(DISTINCT attr) are supported. Currently there can be at most one COUNT(DISTINCT) per query and an argument needs to be an attribute. Both current restrictions on COUNT(DISTINCT) might be lifted in the future. A special GROUPBY() function is also supported. It returns the GROUP BY key. That is particularly useful when grouping by an MVA value, in order to pick the specific value that was used to create the current group.

SELECT *, AVG(price) AS avgprice, COUNT(DISTINCT storeid), GROUPBY()
FROM products
WHERE MATCH('ipod')
GROUP BY vendorid


GROUP BY on a string attribute is supported, with respect for current collation (see Collations).

You can query Manticore to return (no more than) N top matches for each group accordingly to WITHIN GROUP ORDER BY.

SELECT id FROM products GROUP 3 BY category


You can sort the result set by (an alias of) the aggregate value.

SELECT group_id, MAX(id) AS max_id
FROM my_index WHERE MATCH('the')
GROUP BY group_id ORDER BY max_id DESC


#### GROUP_CONCAT()¶

When you group by an attribute, the result set only shows attributes from a single document representing the whole group. GROUP_CONCAT() produces a comma-separated list of the attribute values of all documents in the group.

SELECT id, GROUP_CONCAT(price) as pricesList, GROUPBY() AS name FROM shops GROUP BY shopName;


#### ZONESPANLIST()¶

ZONESPANLIST() function returns pairs of matched zone spans. Each pair contains the matched zone span identifier, a colon, and the order number of the matched zone span. For example, if a document reads <emphasis role=”bold”><i>text</i> the <i>text</i></emphasis>, and you query for ‘ZONESPAN:(i,b) text’, then ZONESPANLIST() will return the string “1:1 1:2 2:1” meaning that the first zone span matched “text” in spans 1 and 2, and the second zone span in span 1 only.

#### WITHIN GROUP ORDER BY¶

This is a Manticore specific extension that lets you control how the best row within a group will to be selected. The syntax matches that of regular ORDER BY clause:

    SELECT *, INTERVAL(posted,NOW()-7*86400,NOW()-86400) AS timeseg, WEIGHT() AS w
FROM example WHERE MATCH('my search query')
GROUP BY siteid
WITHIN GROUP ORDER BY w DESC
ORDER BY timeseg DESC, w DESC

WITHIN GROUP ORDER BY on a string attribute is supported, with
respect for current collation (see :ref:collations).


#### HAVING¶

This is used to filter on GROUP BY values. Currently supports only one filtering condition.

SELECT id FROM plain GROUP BY title HAVING group_id=16;
SELECT id FROM plain GROUP BY attribute HAVING COUNT(*)>1;


Because of HAVING is implemented as a whole result set post-processing, result set for query with HAVING could be less than max_matches allows.

#### ORDER BY¶

Unlike in regular SQL, only column names (not expressions) are allowed and explicit ASC and DESC are required. The columns however can be computed expressions:

SELECT *, WEIGHT()*10+docboost AS skey FROM example ORDER BY skey


You can use subqueries to speed up specific searches, which involve reranking, by postponing hard (slow) calculations as late as possible. For example, SELECT id,a_slow_expression() AS cond FROM an_index ORDER BY id ASC, cond DESC LIMIT 100; could be better written as SELECT * FROM (SELECT id,a_slow_expression() AS cond FROM an_index ORDER BY id ASC LIMIT 100) ORDER BY cond DESC; because in the first case the slow expression would be evaluated for the whole set, while in the second one it would be evaluated just for a subset of values.

ORDER BY on a string attribute is supported, with respect for current collation (see Collations).

ORDER BY RAND() syntax is supported. Note that this syntax is actually going to randomize the weight values and then order matches by those randomized weights.

#### LIMIT¶

Both LIMIT N and LIMIT M,N forms are supported. Unlike in regular SQL (but like in Manticore API), an implicit LIMIT 0,20 is present by default.

#### OPTION¶

This is a Manticore specific extension that lets you control a number of per-query options. The syntax is:

OPTION <optionname>=<value> [ , ... ]


Supported options and respectively allowed values are:

• agent_query_timeout - integer (max time in milliseconds to wait for remote queries to complete, see agent_query_timeout under Index configuration options for details)

• boolean_simplify - 0 or 1, enables simplifying the query to speed it up

• comment - string, user comment that gets copied to a query log file

• cutoff - integer (max found matches threshold)

• field_weights - a named integer list (per-field user weights for ranking)

• global_idf - use global statistics (frequencies) from the global_idf file for IDF computations, rather than the local index statistics.

• idf - a quoted, comma-separated list of IDF computation flags. Known flags are:

• normalized: BM25 variant, idf = log((N-n+1)/n), as per Robertson et al
• plain: plain variant, idf = log(N/n), as per Sparck-Jones
• tfidf_normalized: additionally divide IDF by query word count, so that TF*IDF fits into [0, 1] range
• tfidf_unnormalized: do not additionally divide IDF by query word count

where N is the collection size and n is the number of matched documents.

The historically default IDF (Inverse Document Frequency) in Manticore is equivalent to OPTION idf='normalized,tfidf_normalized', and those normalizations may cause several undesired effects.

First, idf=normalized causes keyword penalization. For instance, if you search for [the | something] and [the] occurs in more than 50% of the documents, then documents with both keywords [the] and [something] will get less weight than documents with just one keyword [something]. Using OPTION idf=plain avoids this. Plain IDF varies in [0, log(N)] range, and keywords are never penalized; while the normalized IDF varies in [-log(N), log(N)] range, and too frequent keywords are penalized.

Second, idf=tfidf_normalized causes IDF drift over queries. Historically, we additionally divided IDF by query keyword count, so that the entire sum(tf*idf) over all keywords would still fit into [0,1] range. However, that means that queries [word1] and [word1 | nonmatchingword2] would assign different weights to the exactly same result set, because the IDFs for both “word1” and “nonmatchingword2” would be divided by 2. OPTION idf='tfidf_unnormalized' fixes that. Note that BM25, BM25A, BM25F() ranking factors will be scale accordingly once you disable this normalization.

IDF flags can be mixed; plain and normalized are mutually exclusive; tfidf_unnormalized and tfidf_normalized are mutually exclusive; and unspecified flags in such a mutually exclusive group take their defaults. That means that OPTION idf=plain is equivalent to a complete OPTION idf='plain,tfidf_normalized' specification.

• local_df - 0 or 1,automatically sum DFs over all the local parts of a distributed index, so that the IDF is consistent (and precise) over a locally sharded index.

• index_weights - a named integer list (per-index user weights for ranking)

• max_matches - integer (per-query max matches value)

Maximum amount of matches that the daemon keeps in RAM for each index and can return to the client. Default is 1000.

Introduced in order to control and limit RAM usage, max_matches setting defines how much matches will be kept in RAM while searching each index. Every match found will still be processed; but only best N of them will be kept in memory and return to the client in the end. Assume that the index contains 2,000,000 matches for the query. You rarely (if ever) need to retrieve all of them. Rather, you need to scan all of them, but only choose “best” at most, say, 500 by some criteria (ie. sorted by relevance, or price, or anything else), and display those 500 matches to the end user in pages of 20 to 100 matches. And tracking only the best 500 matches is much more RAM and CPU efficient than keeping all 2,000,000 matches, sorting them, and then discarding everything but the first 20 needed to display the search results page. max_matches controls N in that “best N” amount.

This parameter noticeably affects per-query RAM and CPU usage. Values of 1,000 to 10,000 are generally fine, but higher limits must be used with care. Recklessly raising max_matches to 1,000,000 means that searchd will have to allocate and initialize 1-million-entry matches buffer for every query. That will obviously increase per-query RAM usage, and in some cases can also noticeably impact performance.

• max_query_time - integer (max search time threshold, msec)

• max_predicted_time - integer (max predicted search time, see predicted_time_costs)

• ranker - any of proximity_bm25, bm25, none, wordcount, proximity, matchany, fieldmask, sph04, expr, or export (refer to Search results ranking for more details on each ranker)

• retry_count - integer (distributed retries count)

• retry_delay - integer (distributed retry delay, msec)

• reverse_scan - 0 or 1, lets you control the order in which full-scan query processes the rows

• sort_method - pq (priority queue, set by default) or kbuffer (gives faster sorting for already pre-sorted data, e.g. index data sorted by id). The result set is in both cases the same; picking one option or the other may just improve (or worsen!) performance.

• rand_seed - lets you specify a specific integer seed value for an ORDER BY RAND() query, for example: … OPTION rand_seed=1234. By default, a new and different seed value is autogenerated for every query.

• low_priority - runs the query with idle priority.

• expand_keywords - 0 or 1, expand keywords with exact forms and/or stars when possible (refer to expand_keywords for more details).

• token_filter - a quoted, colon-separated of library name:plugin name:optional string of settings. Query-time token filter gets created on search each time full-text invoked by every index involved and let you implement a custom tokenizer that makes tokens according to custom rules. SELECT * FROM index WHERE MATCH ('yes@no') OPTION token_filter='mylib.so:blend:@'

• morphology - none, allows to replace all query terms with their exact forms if index was built with index_exact_words enabled. Useful to prevent stemming or lemmatizing query terms.

Example:

SELECT * FROM test WHERE MATCH('@title hello @body world')
OPTION ranker=bm25, max_matches=3000,
field_weights=(title=10, body=3), agent_query_timeout=10000


#### FACET¶

This Manticore specific extension enables faceted search with subtree optimization. It is capable of returning multiple result sets with a single SQL statement, without the need for complicated multi-queries. FACET clauses should be written at the very end of SELECT statements with spaces between them.

FACET {expr_list} [BY {expr_list}] [ORDER BY {expr | FACET()} {ASC | DESC}] [LIMIT [offset,] count]
SELECT * FROM test FACET brand_id FACET categories;
SELECT * FROM test FACET brand_name BY brand_id ORDER BY brand_name ASC FACET property;


Working example:

mysql> SELECT *, IN(brand_id,1,2,3,4) AS b FROM facetdemo WHERE MATCH('Product') AND b=1 LIMIT 0,10
FACET brand_name, brand_id BY brand_id ORDER BY brand_id ASC
FACET property ORDER BY COUNT(*) DESC
FACET INTERVAL(price,200,400,600,800) ORDER BY FACET() ASC
FACET categories ORDER BY FACET() ASC;
+------+-------+----------+-------------------+-------------+----------+------------+------+
| id   | price | brand_id | title             | brand_name  | property | categories | **    |
+------+-------+----------+-------------------+-------------+----------+------------+------+
|    1 |   668 |        3 | Product Four Six  | Brand Three | Three    | 11,12,13   |    1 |
|    2 |   101 |        4 | Product Two Eight | Brand Four  | One      | 12,13,14   |    1 |
|    8 |   750 |        3 | Product Ten Eight | Brand Three | Five     | 13         |    1 |
|    9 |    49 |        1 | Product Ten Two   | Brand One   | Three    | 13,14,15   |    1 |
|   13 |   613 |        1 | Product Six Two   | Brand One   | Eight    | 13         |    1 |
|   20 |   985 |        2 | Product Two Six   | Brand Two   | Nine     | 10         |    1 |
|   22 |   501 |        3 | Product Five Two  | Brand Three | Four     | 12,13,14   |    1 |
|   23 |   765 |        1 | Product Six Seven | Brand One   | Nine     | 11,12      |    1 |
|   28 |   992 |        1 | Product Six Eight | Brand One   | Two      | 12,13      |    1 |
|   29 |   259 |        1 | Product Nine Ten  | Brand One   | Five     | 12,13,14   |    1 |
+------+-------+----------+-------------------+-------------+----------+------------+------+
+-------------+----------+----------+
| brand_name  | brand_id | count(*) |
+-------------+----------+----------+
| Brand One   |        1 |     1012 |
| Brand Two   |        2 |     1025 |
| Brand Three |        3 |      994 |
| Brand Four  |        4 |      973 |
+-------------+----------+----------+
+----------+----------+
| property | count(*) |
+----------+----------+
| One      |      427 |
| Five     |      420 |
| Seven    |      420 |
| Two      |      418 |
| Three    |      407 |
| Six      |      401 |
| Nine     |      396 |
| Eight    |      387 |
| Four     |      371 |
| Ten      |      357 |
+----------+----------+
+---------------------------------+----------+
| interval(price,200,400,600,800) | count(*) |
+---------------------------------+----------+
|                               0 |      799 |
|                               1 |      795 |
|                               2 |      757 |
|                               3 |      833 |
|                               4 |      820 |
+---------------------------------+----------+
+------------+----------+
| categories | count(*) |
+------------+----------+
|         10 |      961 |
|         11 |     1653 |
|         12 |     1998 |
|         13 |     2090 |
|         14 |     1058 |
|         15 |      347 |
+------------+----------+


#### Subselects¶

In format SELECT * FROM (SELECT … ORDER BY cond1 LIMIT X) ORDER BY cond2 LIMIT Y. The outer select allows only ORDER BY and LIMIT clauses. Subselects currently have 2 usage cases:

1. We have a query with 2 ranking UDFs, one very fast and the other one slow and we perform a full-text search will a big match result set. Without subselect the query would look like

SELECT id,slow_rank() as slow,fast_rank() as fast FROM index
WHERE MATCH(‘some common query terms’) ORDER BY fast DESC, slow DESC LIMIT 20
OPTION max_matches=1000;


With subselects the query can be rewritten as :

SELECT * FROM
(SELECT id,slow_rank() as slow,fast_rank() as fast FROM index WHERE
MATCH(‘some common query terms’)
ORDER BY fast DESC LIMIT 100 OPTION max_matches=1000)
ORDER BY slow DESC LIMIT 20;


In the initial query the slow_rank() UDF is computed for the entire match result set. With subselects, only fast_rank() is computed for the entire match result set, while slow_rank() is only computed for a limited set.

2. The second case comes handy for large result set coming from a distributed index.

For this query:

SELECT * FROM my_dist_index WHERE some_conditions LIMIT 50000;


If we have 20 nodes, each node can send back to master a number of 50K records, resulting in 20 x 50K = 1M records, however as the master sends back only 50K (out of 1M), it might be good enough for us for the nodes to send only the top 10K records. With subselect we can rewrite the query as:

SELECT * FROM
(SELECT * FROM my_dist_index WHERE some_conditions LIMIT 10000)
ORDER by some_attr LIMIT 50000;


In this case, the nodes receive only the inner query and execute. This means the master will receive only 20x10K=200K records. The master will take all the records received, reorder them by the OUTER clause and return the best 50K records. The subselect help reducing the traffic between the master and the nodes and also reduce the master’s computation time (as it process only 200K instead of 1M).

### SELECT @@system_variable syntax¶

SELECT @@system_variable [LIMIT [offset,] row_count]


This is currently a placeholder query that does nothing and reports success. That is in order to keep compatibility with frameworks and connectors that automatically execute this statement.

However @@session.last_insert_id and LAST_INSERT_ID() report ID of documents these were inserted or replaced well at last statement.

mysql> select @@session.last_insert_id;
+--------------------------+
| @@session.last_insert_id |
+--------------------------+
| 11,32                    |
+--------------------------+
1 rows in set

mysql> select LAST_INSERT_ID();
+------------------+
| LAST_INSERT_ID() |
+------------------+
| 25,26,29         |
+------------------+
1 rows in set


### SET syntax¶

SET [GLOBAL] server_variable_name = value
SET [INDEX index_name] GLOBAL @user_variable_name = (int_val1 [, int_val2, ...])
SET NAMES value [COLLATE value]
SET @@dummy_variable = ignored_value


SET statement modifies a variable value. The variable names are case-insensitive. No variable value changes survive server restart.

SET NAMES statement and SET @@variable_name syntax, both introduced do nothing. They were implemented to maintain compatibility with 3rd party MySQL client libraries, connectors, and frameworks that may need to run this statement when connecting.

There are the following classes of the variables:

1. per-session server variable
2. global server variable
3. global user variable
4. global distributed variable

Global user variables are shared between concurrent sessions. Currently, the only supported value type is the list of BIGINTs, and these variables can only be used along with IN() for filtering purpose. The intended usage scenario is uploading huge lists of values to searchd (once) and reusing them (many times) later, saving on network overheads. Global user variables might be either transferred to all agents of distributed index or set locally in case of local index defined at distibuted index. Example:

// in session 1
mysql> SET GLOBAL @myfilter=(2,3,5,7,11,13);
Query OK, 0 rows affected (0.00 sec)

// later in session 2
mysql> SELECT * FROM test1 WHERE group_id IN @myfilter;
+------+--------+----------+------------+-----------------+------+
| id   | weight | group_id | date_added | title           | tag  |
+------+--------+----------+------------+-----------------+------+
|    3 |      1 |        2 | 1299338153 | another doc     | 15   |
|    4 |      1 |        2 | 1299338153 | doc number four | 7,40 |
+------+--------+----------+------------+-----------------+------+
2 rows in set (0.02 sec)


Per-session and global server variables affect certain server settings in the respective scope. Known per-session server variables are:

• AUTOCOMMIT = {0 | 1} Whether any data modification statement should be implicitly wrapped by BEGIN and COMMIT.
• COLLATION_CONNECTION = collation_name Selects the collation to be used for ORDER BY or GROUP BY on string values in the subsequent queries. Refer to Collations for a list of known collation names.
• CHARACTER_SET_RESULTS = charset_name Does nothing; a placeholder to support frameworks, clients, and connectors that attempt to automatically enforce a charset when connecting to a Manticore server.
• SQL_AUTO_IS_NULL = value Does nothing; a placeholder to support frameworks, clients, and connectors that attempt to automatically enforce a charset when connecting to a Manticore server.
• SQL_MODE = value Does nothing; a placeholder to support frameworks, clients, and connectors that attempt to automatically enforce a charset when connecting to a Manticore server.
• WAIT_TIMEOUT = value Does nothing; added for improved compatibility with 3rd party MySQL clients
• PROFILING = {0 | 1} Enables query profiling in the current session. Defaults to 0. See also SHOW PROFILE syntax.

Known global server variables are:

• QUERY_LOG_FORMAT = {plain | sphinxql} Changes the current log format.
• LOG_LEVEL = {info | debug | replication | debugv | debugvv} Changes the current log verboseness level.
• QCACHE_MAX_BYTES = <value> Changes the query cache RAM use limit to a given value.
• QCACHE_THRESH_MSEC = <value> Changes the query cache minimum wall time threshold to a given value.
• QCACHE_TTL_SEC = <value> Changes the query cache TTL for a cached result to a given value.
• MAINTENANCE = {0 | 1} When set to 1, puts the server in maintenance mode. Only clients with vip connections can execute queries in this mode. All new non-vip incoming connections are refused.
• GROUPING_IN_UTC = {0 | 1} When set to 1, cause timed grouping functions (day(), month(), year(), yearmonth(), yearmonthday()) to be calculated in utc. Read the doc for grouping_in_utc config params for more details.

Examples:

mysql> SET autocommit=0;
Query OK, 0 rows affected (0.00 sec)

mysql> SET GLOBAL query_log_format=sphinxql;
Query OK, 0 rows affected (0.00 sec)


### SET TRANSACTION syntax¶

SET TRANSACTION ISOLATION LEVEL { READ UNCOMMITTED
| SERIALIZABLE }


SET TRANSACTION statement does nothing. It was implemented to maintain compatibility with 3rd party MySQL client libraries, connectors, and frameworks that may need to run this statement when connecting.

Example:

mysql> SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
Query OK, 0 rows affected (0.00 sec)


### SHOW AGENT STATUS¶

SHOW AGENT ['agent'|'index'] STATUS [ LIKE pattern ]


Displays the statistic of remote agents or distributed index. It includes the values like the age of the last request, last answer, the number of different kind of errors and successes, etc. The statistic is shown for every agent for last 1, 5 and 15 intervals, each of them of ha_period_karma seconds. The command exists only in sphinxql.

mysql> SHOW AGENT STATUS;
+------------------------------------+----------------------------+
| Variable_name                      | Value                      |
+------------------------------------+----------------------------+
| status_period_seconds              | 60                         |
| status_stored_periods              | 15                         |
| ag_0_hostname                      | 192.168.0.202:6713         |
| ag_0_references                    | 2                          |
| ag_0_lastquery                     | 0.41                       |
| ag_0_lastperiodmsec                | 222                        |
| ag_0_errorsarow                    | 0                          |
| ag_0_1periods_query_timeouts       | 0                          |
| ag_0_1periods_connect_timeouts     | 0                          |
| ag_0_1periods_connect_failures     | 0                          |
| ag_0_1periods_network_errors       | 0                          |
| ag_0_1periods_wrong_replies        | 0                          |
| ag_0_1periods_unexpected_closings  | 0                          |
| ag_0_1periods_warnings             | 0                          |
| ag_0_1periods_succeeded_queries    | 27                         |
| ag_0_1periods_msecsperquery        | 232.31                     |
| ag_0_5periods_query_timeouts       | 0                          |
| ag_0_5periods_connect_timeouts     | 0                          |
| ag_0_5periods_connect_failures     | 0                          |
| ag_0_5periods_network_errors       | 0                          |
| ag_0_5periods_wrong_replies        | 0                          |
| ag_0_5periods_unexpected_closings  | 0                          |
| ag_0_5periods_warnings             | 0                          |
| ag_0_5periods_succeeded_queries    | 146                        |
| ag_0_5periods_msecsperquery        | 231.83                     |
| ag_1_hostname                      | 192.168.0.202:6714         |
| ag_1_references                    | 2                          |
| ag_1_lastquery                     | 0.41                       |
| ag_1_lastperiodmsec                | 220                        |
| ag_1_errorsarow                    | 0                          |
| ag_1_1periods_query_timeouts       | 0                          |
| ag_1_1periods_connect_timeouts     | 0                          |
| ag_1_1periods_connect_failures     | 0                          |
| ag_1_1periods_network_errors       | 0                          |
| ag_1_1periods_wrong_replies        | 0                          |
| ag_1_1periods_unexpected_closings  | 0                          |
| ag_1_1periods_warnings             | 0                          |
| ag_1_1periods_succeeded_queries    | 27                         |
| ag_1_1periods_msecsperquery        | 231.24                     |
| ag_1_5periods_query_timeouts       | 0                          |
| ag_1_5periods_connect_timeouts     | 0                          |
| ag_1_5periods_connect_failures     | 0                          |
| ag_1_5periods_network_errors       | 0                          |
| ag_1_5periods_wrong_replies        | 0                          |
| ag_1_5periods_unexpected_closings  | 0                          |
| ag_1_5periods_warnings             | 0                          |
| ag_1_5periods_succeeded_queries    | 146                        |
| ag_1_5periods_msecsperquery        | 230.85                     |
+------------------------------------+----------------------------+
50 rows in set (0.01 sec)


An optional LIKE clause is supported. Refer to SHOW META syntax for its syntax details.

mysql> SHOW AGENT STATUS LIKE '%5period%msec%';
+-----------------------------+--------+
| Key                         | Value  |
+-----------------------------+--------+
| ag_0_5periods_msecsperquery | 234.72 |
| ag_1_5periods_msecsperquery | 233.73 |
| ag_2_5periods_msecsperquery | 343.81 |
+-----------------------------+--------+
3 rows in set (0.00 sec)


You can specify a particular agent by its address. In this case only that agent’s data will be displayed. Also, agent_ prefix will be used instead of ag_N_:

mysql> SHOW AGENT '192.168.0.202:6714' STATUS LIKE '%15periods%';
+-------------------------------------+--------+
| Variable_name                       | Value  |
+-------------------------------------+--------+
| agent_15periods_query_timeouts      | 0      |
| agent_15periods_connect_timeouts    | 0      |
| agent_15periods_connect_failures    | 0      |
| agent_15periods_network_errors      | 0      |
| agent_15periods_wrong_replies       | 0      |
| agent_15periods_unexpected_closings | 0      |
| agent_15periods_warnings            | 0      |
| agent_15periods_succeeded_queries   | 439    |
| agent_15periods_msecsperquery       | 231.73 |
+-------------------------------------+--------+
9 rows in set (0.00 sec)


Finally, you can check the status of the agents in a specific distributed index. It can be done with a SHOW AGENT ‘index’ STATUS statement. That statement shows the index HA status (ie. whether or not it uses agent mirrors at all), and then the mirror information (specifically: address, blackhole and persistent flags, and the mirror selection probability used when one of the weighted-probability strategies is in effect).

mysql> SHOW AGENT dist_index STATUS;
+--------------------------------------+--------------------------------+
| Variable_name                        | Value                          |
+--------------------------------------+--------------------------------+
| dstindex_1_is_ha                     | 1                              |
| dstindex_1mirror1_id                 | 192.168.0.202:6713:loc         |
| dstindex_1mirror1_probability_weight | 0.372864                       |
| dstindex_1mirror1_is_blackhole       | 0                              |
| dstindex_1mirror1_is_persistent      | 0                              |
| dstindex_1mirror2_id                 | 192.168.0.202:6714:loc         |
| dstindex_1mirror2_probability_weight | 0.374635                  `