Installation¶
Installing Manticore packages on Debian and Ubuntu¶
Supported releases:
Debian
- 7.0 (wheezy)
- 8.0 (jessie)
Ubuntu
- 14.04 LTS (trusty)
- 16.05 LTS (xenial)
Supported platforms:
- x86
- x86_64
Manticore requires a few libraries to be installed on Debian/Ubuntu. Use apt-get to download and install these dependencies:
$ sudo apt-get install mysql-client unixodbc libpq5
Now you can install Manticore:
$ wget https://github.com/manticoresoftware/manticore/releases/download/2.3.3-170706/manticore_2.3.3-170706-9fbdd5f-1.jessie_amd64.deb
$ sudo dpkg -i manticore_2.3.3-170706-9fbdd5f-1.jessie_amd64.deb
After preparing configuration file (see Quick tour), you can start searchd daemon:
$ systemctl manticore start
Installing Manticore packages on RedHat and CentOS¶
Supported releases:
- CentOS 6 and RHEL 6
- CentOS 7 and RHEL 7
Supported platforms:
- x86
- x86_64
Before installation make sure you have these packages installed:
$ yum install mysql-client postgresql-libs unixODBC
Download RedHat RPM from Manticore website and install it:
$ wget https://github.com/manticoresoftware/manticore/releases/download/2.3.3-170706/manticore-2.3.3.170706.9fbdd5f-1rhel7.x86_64.rpm
$ rpm -Uhv manticore-2.3.3.170706.9fbdd5f-1rhel7.x86_64.rpm
After preparing configuration file (see Quick tour), you can start searchd daemon:
$ systemctl searchd start
Installing Manticore on Windows¶
To install on Windows, you need to download the zip package and unpack it first.
cd C:\Manticore
unzip manticore-2.3.3-170706-9fbdd5f-dev-pgsql-stemmer-x64-bin.zip
Edit the contents of sphinx.conf.in - specifically entries relating to @CONFDIR@ - to paths suitable for your system.
Install the searchd
system as a Windows service:
C:\Manticore\bin> C:\Manticore\bin\searchd --install --config C:\Manticore\sphinx.conf.in --servicename Manticore
- The
searchd
service will now be listed in the Services panel within the Management Console, available from Administrative Tools. It will not have been started, as you will need to configure it and build your indexes withindexer
before starting the service. A guide to do this can be found under Quick tour.
Compiling Manticore from source¶
Required tools¶
a working compiler
- on Linux - GNU gcc (4.7.2 and above) and clang can be used
- on Windows - Microsoft Visual Studio (community version is enough)
- on Mac OS - XCode
cmake - used on all plaftorms
Optional dependencies¶
- git, flex, bison - needed if the sources are from cloned repository and not the source tarball
- development version of MySQL client for MySQL source driver
- development version of unixODBC for the unixODBC source driver
- development version of libPQ for the PostgreSQL source driver
- development version of libexpat for the XMLpipe source driver
- RE2 (bundled in the source tarball) for regexp_filter feature
- lib stemmer (bundled in the source tarball ) for additional language stemmers
General building options¶
For compiling latest version of Manticore, recommended is checkout the latest code from the github repositiory. Alternative, for compiling a certain version, you can either checked that version from github or use it’s respective source tarball.
$ git clone https://github.com/manticoresoftware/manticore.git
$ wget https://github.com/manticoresoftware/manticore/releases/download/2.3.3-170706/manticore-2.3.3.170706.9fbdd5f.tar.gz
$ tar zcvf manticore-2.3.3.170706.9fbdd5f.tar.gz
Manticore uses cmake for building sources. We recommend using a folder outside the sources for the building workspace to keep clean the source folders.
$ mkdir build
$ cd build
Next step is to configure the building with cmake. Available list of configuration options:
CMAKE_BUILD_TYPE
- can be Debug , Release , MinSizeRel and RelWithDebInfo (default).SPLIT_SYMBOLS
(bool) - specify whenever to create separate files with debugging symbols. In the default build type,RelWithDebInfo, the binaries include the debug symbols. With this option specified, the binaries will be stripped of the debug symbols , which will be put in separate filesUSE_BISON,USE_FLEX
(bool) - enabled by default, specifies whenever to enable bison and flex toolsLIBS_BUNDLE
- filepath to a folder with different libraries. This is mostly relevant for Windows buildingWITH_STEMMER
(bool) - specifies if the build should include the libstemmer library. The library is searched in several places, starting with- libstemmer_c folder in the source directory
- common system path. Please note that in this case, the linking is dynamic and libstemmer should be available system-wide on the installed systems
- libstemmer_c.tgz in
LIBS_BUNDLE
folder. - download from snowball project website. This is done by cmake and no additional tool is required
WITH_RE2
(bool) - specifies if the build should include the RE2 library. The library can be taken from the following locations:- in the folder specified by
WITH_RE2_ROOT
parameters - in libre2 folder of the Manticore sources
- system wide search, while first looking for headers specified by
WITH_RE2_INCLUDES
folder and the lib files inWITH_RE2_LIBS
folder - check presence of master.zip in the
LIBS_BUNDLE
folder - Download from https://github.com/manticoresoftware/re2/archive/master.zip
- in the folder specified by
WITH_EXPAT
(bool) enabled compiling with libexpat, used XMLpipe source driverWITH_MYSQL
(bool) enabled compiling with MySQL client library, used by MySQL source driver. Additional parametersWITH_MYSQL_ROOT
,WITH_MYSQL_LIBS
andWITH_MYSQL_INCLUDES
can be used for custom MySQL filesWITH_ODBC
(bool) enabled compiling with ODBC client library, used by ODBC source driverWITH_PGSQL
(bool) enabled compiling with PostgreSQL client library, used by PostgreSQL source driverDISTR_BUILD
- in case the target is packaging, it specifies the target operating system. Supported values are: centos6, centos7, wheezy, jessie, trusty, xenial, macos
Compiling on UNIX systems¶
To install all dependencies on Debian/Ubuntu:
$ apt-get install build-essential cmake unixodbc-dev libpq-dev libexpat-dev libmysqlclient-dev git flex bison
To install all dependencies on CentOS/RHEL:
$ yum install gcc gcc-c++ make cmake mysql-devel expat-devel postgresql-devel unixODBC-devel rpm-build systemd-units git flex bison
RHEL/CentOS 6 ship with a old version of the gcc compiler, which doesn’t support std-c++ standard, for compiling use devtools repository:
$ wget http://people.centos.org/tru/devtools-2/devtools-2.repo -O /etc/yum.repos.d/devtools-2.repo
$ yum upgrade -y
$ yum install -y devtoolset-2-gcc devtoolset-2-binutils devtoolset-2-gcc-c++
$ export PATH=/opt/rh/devtoolset-2/root/usr/bin:$PATH
We recommend using a separate folder for building instead of compiling in the source folder:
$ mkdir build
$ cd build
$ cmake -D WITH_MYSQL=TRUE -DWITH_RE2=1 ../manticore
or if we use sources from tarball:
$ cmake -D WITH_MYSQL=TRUE -DWITH_RE2=1 ../manticore-2.3.3.170706.9fbdd5f
To simply compile:
$ make -j4
This will create the binary files, however we want to either install Manticore or more convenient to create a package. To install just do
$ make -j4 install
For packaging use package
$ make -j4 package
By default, if no operating system was targeted, package
will create only a zip with the binaries.
If, for example, we want to create a deb package for Debian Jessie, we need to specify to cmake the DISTR_BUILD
parameter:
$ cmake -D WITH_MYSQL=TRUE -DWITH_RE2=1 -DDISTR_BUILD=jessie../manticore
$ make -j4 package
This will create 2 deb packages, a manticore-x.x.x-bin.deb and a manticore-x.x.x-dbg.deb which contains the version with debug symbols.
Another possible target is tarball
, which create a tar.gz file from the sources.
Compiling on Windows¶
For building on Windows you need:
- Visual Studio
- Cmake for Windows
- Expat, MySQL and PostgreSQL in bundle directory.
For a simple building on x64:
C:\build>"%PROGRAMW6432%\CMake\bin\cmake.exe" -G "Visual Studio 14 Win64" -DLIBS_BUNDLE="C:\bundle" -DDISTR=none -DCPACK_GENERATOR=ZIP "C:\manticore"
C:\build>"%PROGRAMW6432%\CMake\bin\cmake.exe" -DWITH_PGSQL=1 -DWITH_RE2=1 -DWITH_STEMMER=1 .
C:\build>"%PROGRAMW6432%\CMake\bin\cmake.exe" --build . --target package --config RelWithDebInfo
Quick Manticore usage tour¶
We are going to use SphinxQL protocol as it’s the current recommended way and it’s also easy to play with. First we connect to Manticore with the normal MySQL client:
$ mysql -h0 -P9306
The default configuration comes with a sample Real-Time. A first step to see it in action is to add several documents to it, then you can start perform searches:
mysql> INSERT INTO rt VALUES (1, 'this is', 'a sample text', 11);
Query OK, 1 row affected (0.00 sec)
mysql> INSERT INTO rt VALUES (2, 'some more', 'text here', 22);
Query OK, 1 row affected (0.00 sec)
mysql> INSERT INTO rt VALUES (3, 'more about this text', 'can be found in this text', 22);
Query OK, 1 row affected (0.00 sec)
mysql> SELECT *,weight() FROM rt WHERE MATCH('text') ORDER BY WEIGHT() DESC;
+------+------+----------+
| id | gid | weight() |
+------+------+----------+
| 3 | 22 | 2252 |
| 1 | 11 | 1319 |
| 2 | 22 | 1319 |
+------+------+----------+
3 rows in set (0.00 sec)
In the sample configuration there is also a plain index with MySQL source, which needs to be indexed first in order to start using it. First, we populate the sample table in MySQL:
mysql> create database test;
$ mysql -u test < /usr/share/doc/manticore/example-conf/example.sql
The sample config uses a test
with no password for connecting to MySQL. Adjust the credentials, then index:
$ sudo -u manticore indexer -c /etc/sphinxsearch/sphinx.conf test1 --rotate
Manticore 2.3.3 9b7033e@170806 master...origin/master-id64-dev
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017, Manticore Software LTD (http://manticoresearch.com)
using config file '/etc/sphinxsearch/sphinx.conf'...
indexing index 'test1'...
collected 4 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 4 docs, 193 bytes
total 0.002 sec, 81503 bytes/sec, 1689.18 docs/sec
total 4 reads, 0.000 sec, 8.1 kb/call avg, 0.0 msec/call avg
total 12 writes, 0.000 sec, 0.1 kb/call avg, 0.0 msec/call avg
rotating indices: successfully sent SIGHUP to searchd (pid=2947).
Now let’s run several queries:
mysql> SELECT *, WEIGHT() FROM test1 WHERE MATCH('"document one"/1');SHOW META;
+------+----------+------------+----------+
| id | group_id | date_added | weight() |
+------+----------+------------+----------+
| 1 | 1 | 1502280778 | 2663 |
| 2 | 1 | 1502280778 | 1528 |
+------+----------+------------+----------+
2 rows in set (0.00 sec)
+---------------+----------+
| Variable_name | Value |
+---------------+----------+
| total | 2 |
| total_found | 2 |
| time | 0.000 |
| keyword[0] | document |
| docs[0] | 2 |
| hits[0] | 2 |
| keyword[1] | one |
| docs[1] | 1 |
| hits[1] | 2 |
+---------------+----------+
9 rows in set (0.00 sec)
mysql> SET profiling=1;SELECT * FROM test1 WHERE id IN (1,2,4);SHOW PROFILE;
Query OK, 0 rows affected (0.00 sec)
+------+----------+------------+
| id | group_id | date_added |
+------+----------+------------+
| 1 | 1 | 1502280778 |
| 2 | 1 | 1502280778 |
| 4 | 2 | 1502280778 |
+------+----------+------------+
3 rows in set (0.00 sec)
+--------------+----------+----------+---------+
| Status | Duration | Switches | Percent |
+--------------+----------+----------+---------+
| unknown | 0.000059 | 4 | 44.70 |
| net_read | 0.000001 | 1 | 0.76 |
| local_search | 0.000042 | 1 | 31.82 |
| sql_parse | 0.000012 | 1 | 9.09 |
| fullscan | 0.000001 | 1 | 0.76 |
| finalize | 0.000007 | 1 | 5.30 |
| aggregate | 0.000006 | 2 | 4.55 |
| net_write | 0.000004 | 1 | 3.03 |
| eval_post | 0.000000 | 1 | 0.00 |
| total | 0.000132 | 13 | 0 |
+--------------+----------+----------+---------+
10 rows in set (0.00 sec)
mysql> SELECT id, id%3 idd FROM test1 WHERE MATCH('this is | nothing') GROUP BY idd;SHOW PROFILE;
+------+------+
| id | idd |
+------+------+
| 1 | 1 |
| 2 | 2 |
| 3 | 0 |
+------+------+
3 rows in set (0.00 sec)
+--------+----------+----------+---------+
| Status | Duration | Switches | Percent |
+--------+----------+----------+---------+
| total | 0.000000 | 0 | 0 |
+--------+----------+----------+---------+
1 row in set (0.00 sec)
mysql> SELECT id FROM test1 WHERE MATCH('is this a good plan?');SHOW PLAN\G
Empty set (0.00 sec)
*************************** 1. row ***************************
Variable: transformed_tree
Value: AND(
AND(KEYWORD(is, querypos=1)),
AND(KEYWORD(this, querypos=2)),
AND(KEYWORD(a, querypos=3)),
AND(KEYWORD(good, querypos=4)),
AND(KEYWORD(plan, querypos=5)))
1 row in set (0.00 sec)
mysql> SELECT COUNT(*) c, id%3 idd FROM test1 GROUP BY idd HAVING COUNT(*)>1;
+------+------+
| c | idd |
+------+------+
| 2 | 1 |
+------+------+
1 row in set (0.00 sec)
mysql> SELECT COUNT(*) FROM test1;
+----------+
| count(*) |
+----------+
| 4 |
+----------+
1 row in set (0.00 sec)
mysql> CALL KEYWORDS ('one two three', 'test1', 1);
+------+-----------+------------+------+------+
| qpos | tokenized | normalized | docs | hits |
+------+-----------+------------+------+------+
| 1 | one | one | 1 | 2 |
| 2 | two | two | 1 | 2 |
| 3 | three | three | 0 | 0 |
+------+-----------+------------+------+------+
3 rows in set (0.00 sec)