Release notes

Version 2.7.5 GA, 4 December 2018

Improvements

  • REGEX function
  • limit/offset for json API search
  • profiler points for qcache

Bugfixes

  • eb3c768 fixed crash of daemon on FACET with multiple attribute wide types
  • d915cf6 fixed implicit group by at main select list of FACET query
  • 5c25dc2 fixed crash on query with GROUP N BY
  • 85d30a2 fixed deadlock on handling crash at memory operations
  • 85166b5 fixed indextool memory consumption during check
  • 58fb031 fixed gmock include not needed anymore as upstream resolve itself

Version 2.7.4 GA, 1 November 2018

Improvements

  • SHOW THREADS in case of remote distributed indexes prints the original query instead of API call
  • SHOW THREADS new option format=sphinxql prints all queries in SphinxQL format
  • SHOW PROFILE prints additional clone_attrs stage

Bugfixes

  • 4f15571 fixed failed to build with libc without malloc_stats, malloc_trim
  • f974f20 fixed special symbols inside words for CALL KEYWORDS result set
  • 0920832 fixed broken CALL KEYWORDS to distributed index via API or to remote agent
  • fd686bf fixed distributed index agent_query_timeout propagate to agents as max_query_time
  • 4ffa623 fixed total documents counter at disk chunk got affected by OPTIMIZE command and breaks weight calculation
  • dcaf4e0 fixed multiple tail hits at RT index from blended
  • eee3817 fixed deadlock at rotation

Version 2.7.3 GA, 26 September 2018

Improvements

  • sort_mode option for CALL KEYWORDS
  • DEBUG on VIP connection can perform ‘crash <password>’ for intentional SIGEGV action on daemon
  • DEBUG can perform ‘malloc_stats’ for dumping malloc stats in searchd.log ‘malloc_trim’ to perform a malloc_trim()
  • improved backtrace is gdb is present on the system

Bugfixes

  • 0f3cc33 fixed crash or hfailure of rename on Windows
  • 1455ba2 fixed crashes of daemon on 32-bit systems
  • ad3710d fixed crash or hung of daemon on empty SNIPPET expression
  • b36d792 fixed broken non progressive optimize and fixed progressive optimize to not create kill-list for oldest disk chunk
  • 34b0324 fixed queue_max_length bad reply for SphinxQL and API at thread pool worker mode
  • ae4b320 fixed crash on adding full-scan query to PQ index with regexp or rlp options set
  • f80f8d5 fixed crash when call one PQ after another
  • 9742f5f refactor AquireAccum
  • 39e5bc3 fixed leak of memory after call pq
  • 21bcc6d cosmetic refactor (c++11 style c-trs, defaults, nullptrs)
  • 2d69039 fixed memory leak on trying to insert duplicate into PQ index
  • 5ed92c4 fixed crash on JSON field IN with large values
  • 4a5262e fixed crash of daemon on CALL KEYWORDS statement to RT index with expansion limit set
  • 552646b fixed invalid filter at PQ matches query;
  • 204f521 introduce small obj allocator for ptr attrs
  • 25453e5 refactor ISphFieldFilter to refcounted flavour
  • 1366ee0 fixed ub/sigsegv when using strtod on non-terminated strings
  • 94bc6fc fixed memory leak in json resultset processing
  • e78e9c9 fixed read over the end of mem block applying attribute add
  • fad572f fixed refactor CSphDict for refcount flavour
  • fd841a4 fixed leak of AOT internal type outside
  • 5ee7f20 fixed memory leak tokenizer management
  • 116c5f1 fixed memory leak in grouper
  • 56fdbc9 special free/copy for dynamic ptrs in matches (memory leak grouper)
  • b1fc161 fixed memory leak of dynamic strings for RT
  • 517b9e8 refactor grouper
  • b1fc161 minor refactor (c++11 c-trs, some reformats)
  • 7034e07 refactor ISphMatchComparator to refcounted flavour
  • b1fc161 privatize cloner
  • efbc051 simplify native little-endian for MVA_UPSIZE, DOCINFO2ID_T, DOCINFOSETID
  • 6da0df4 add valgrind support to to ubertests
  • 1d17669 fixed crash because race of ‘success’ flag on connection
  • 5a09c32 switch epoll to edge-triggered flavour
  • 5d52868 fixed IN statement in expression with formatting like at filter
  • bd8b3c9 fixed crash at RT index on commit of document with large docid
  • ce656b8 fixed argless options in indextool
  • 08c9507 fixed memory leak of expanded keyword
  • 30c75a2 fixed memory leak of json grouper
  • 6023f26 fixed leak of global user vars
  • 7c138f1 fixed leakage of dynamic strings on early rejected matches
  • 9154b18 fixed leakage on length(<expression>)
  • 43fca3a fixed memory leak because strdup() in parser
  • 71ff777 fixed refactor expression parser to accurate follow refcounts

Version 2.7.2 GA, 27 August 2018

Improvements

  • compatibility with MySQL 8 clients
  • TRUNCATE WITH RECONFIGURE
  • retired memory counter on SHOW STATUS for RT indexes
  • global cache of multi agents
  • improved IOCP on Windows
  • VIP connections for HTTP protocol
  • SphinxQL DEBUG command which can run various subcommands
  • shutdown_token - SHA1 hash of password needed to invoke shutdown using DEBUG command
  • new stats to SHOW AGENT STATUS (_ping, _has_perspool, _need_resolve)
  • –verbose option of indexer now accept [debug|debugv|debugvv] for printing debug messages

Bugfixes

  • 390082 removed wlock at optimize
  • 4c3376 fixed wlock at reload index settings
  • b5ea8d fixed memory leak on query with JSON filter
  • 930e83 fixed empty documents at PQ result set
  • 53deec fixed confusion of tasks due to removed one
  • cad9b9 fixed wrong remote host counting
  • 90008c fixed memory leak of parsed agent descriptors
  • 978d83 fixed leak in search
  • 019394 cosmetic changes on explicit/inline c-trs, override/final usage
  • 943e29 fixed leak of json in local/remote schema
  • 02dbdd fixed leak of json sorting col expr in local/remote schema
  • c74d0b fixed leak of const alias
  • 6e5b57 fixed leak of preread thread
  • 39c740 fixed stuck on exit because of stucked wait in netloop
  • adaf97 fixed stuck of ‘ping’ behaviour on change HA agent to usual host
  • 32c40e separate gc for dashboard storage
  • 511a3c fixed ref-counted ptr fix
  • 32c40e fixed indextool crash on unexistent index
  • 156edc fixed output name of exceeding attr/field in xmlpipe indexing
  • cdac6d fixed default indexer’s value if no indexer section in config
  • e61ec0 fixed wrong embedded stopwords in disk chunk by RT index after daemon restart
  • 5fba49 fixed skip phantom (already closed, but not finally deleted from the poller) connections
  • f22ae3 fixed blended (orphaned) network tasks
  • 46890e fixed crash on read action after write
  • 03f9df fixed searchd crashes when running tests on windows
  • e9255e fixed handle EINPROGRESS code on usual connect()
  • 248b72 fixed connection timeouts when working with TFO

Version 2.7.1 GA, 4 July 2018

Improvements

  • improved wildcards performance on matching multiple documents at PQ
  • support for fullscan queries at PQ
  • support for MVA attributes at PQ
  • regexp and RLP support for percolate indexes

Bugfixes

  • 688562 fixed loose of query string
  • 0f1770 fixed empty info at SHOW THREADS statement
  • 53faa3 fixed crash on matching with NOTNEAR operator
  • 26029a fixed error message on bad filter to PQ delete

Version 2.7.0 GA, 11 June 2018

Improvements

  • reduced number of syscalls to avoid Meltdown and Spectre patches impact
  • internal rewrite of local index management
  • remote snippets refactor
  • full configuration reload
  • all node connections are now independent
  • proto improvements
  • Windows communication switched from wsapoll to IO completion ports
  • TFO can be used for communication between master and nodes
  • SHOW STATUS now outputs to daemon version and mysql_version_string
  • added docs_id option for documents called in CALL PQ.
  • percolate queries filter can now contain expressions
  • distributed indexes can work with FEDERATED
  • dummy SHOW NAMES COLLATE and SET wait_timeout (for better ProxySQL compatibility)

Bugfixes

  • 5bcff0 fixed added not equal to tags of PQ
  • 9ebc58 fixed added document id field to JSON document CALL PQ statement
  • 8ae0e5 fixed flush statement handlers to PQ index
  • c24b15 fixed PQ filtering on JSON and string attributes
  • 1b8bdd fixed parsing of empty JSON string
  • 1ad8a0 fixed crash at multi-query with OR filters
  • 69b898 fixed indextool to use config common section (lemmatizer_base option) for commands (dumpheader)
  • 6dbeaf fixed empty string at result set and filter
  • 39c4eb fixed negative document id values
  • 266b70 fixed word clip length for very long words indexed
  • 47823b fixed matching multiple documents of wildcard queries at PQ

Version 2.6.4 GA, 3 May 2018

Features and improvements

  • MySQL FEDERATED engine support
  • MySQL packets return now SERVER_STATUS_AUTOCOMMIT flag, adds compatibility with ProxySQL
  • listen_tfo - enable TCP Fast Open connections for all listeners
  • indexer –dumpheader can dump also RT header from .meta file
  • cmake build script for Ubuntu Bionic

Bugfixes

  • 355b116 fixed invalid query cache entries for RT index;
  • 546e229 fixed index settings got lost next after seamless rotation
  • 0c45098 fixed fixed infix vs prefix length set; added warning on unsupported infix length
  • 80542fa fixed RT indexes auto-flush order
  • 705d8c5 fixed result set schema issues for index with multiple attributes and queries to multiple indexes
  • b0ba932 fixed some hits got lost at batch insert with document duplicates
  • 4510fa4 fixed optimize failed to merge disk chunks of RT index with large documents count

Version 2.6.3 GA, 28 March 2018

Improvements

  • jemalloc at compilation. If jemalloc is present on system, it can be enabled with cmake flag -DUSE_JEMALLOC=1

Bugfixes

  • 85a6d7e fixed log expand_keywords option into sphinxql query log
  • caaa384 fixed HTTP interface to correctly process query with large size
  • e386d84 fixed crash of daemon on DELETE to RT index with index_field_lengths enable
  • cd538f3 fixed cpustats searchd cli option to work with unsupported systems
  • 8740fd6 fixed utf8 substring matching with min lengths defined

Version 2.6.2 GA, 23 February 2018

Improvements

  • improved Percolate Queries performance in case of using NOT operator and for batched documents.
  • CALL PQ can use multiple threads depending on dist_threads
  • new full-text matching operator NOTNEAR/N
  • LIMIT for SELECT on percolate indexes
  • expand_keywords can accept ‘start’,’exact’ (where ‘star,exact’ has same effect as ‘1’)
  • ranged-main-query for joined fields which uses the ranged query defined by sql_query_range

Bugfixes

  • 72dcf66 fixed crash on searching ram segments; deadlock on save disk chunk with double buffer; deadlock on save disk chunk during optimize
  • 3613714 fixed indexer crash on xml embeded schema with empty attribute name
  • 48d7e80 fixed erroneous unlinking of not-owned pid-file
  • a5563a4 fixed orphaned fifos sometimes left in temp folder
  • 2376e8f fixed empty FACET result set with wrong NULL row
  • 4842b67 fixed broken index lock when running daemon as windows service
  • be35fee fixed wrong iconv libs on mac os
  • 83744a9 fixed wrong count(*)

Version 2.6.1 GA, 26 January 2018

Improvements

  • agent_retry_count in case of agents with mirrors gives the value of retries per mirror instead of per agent, the total retries per agent being agent_retry_count*mirrors.
  • agent_retry_count can now be specified per index, overriding global value. An alias mirror_retry_count is added.
  • a retry_count can be specified in agent definition and the value represents retries per agent
  • Percolate Queries are now in HTTP JSON API at /json/pq.
  • Added -h and -v options (help and version) to executables
  • morphology_skip_fields support for Real-Time indexes

Bugfixes

  • a40b079 fixed ranged-main-query to correctly work with sql_range_step when used at MVA field
  • f2f5375 fixed issue with blackhole system loop hung and blackhole agents seems disconnected
  • 84e1f54 fixed query id to be consistent, fixed duplicated id for stored queries
  • 1948423 fixed daemon crash on shutdown from various states
  • 9a706b 3495fd7 timeouts on long queries
  • 3359bcd8 refactored master-agent network polling on kqueue-based systems (Mac OS X, BSD).

Version 2.6.0, 29 December 2017

Features and improvements

Bugfixes

  • 0cfae4c fixed crash on debug build of daemon (and m.b. UB on release) when built with rlp
  • 324291e fixed RT index optimize with progressive option enabled that merges kill-lists with wrong order
  • ac0efee minor crash on mac
  • lots of minor fixes after thorough static code analysis
  • other minor bugfixes

Upgrade

In this release we’ve changed internal protocol used by masters and agents to speak with each other. In case you run Manticoresearch in a distributed environment with multiple instances make sure your first upgrade agents, then the masters.

Version 2.5.1, 23 November 2017

Features and improvements

  • JSON queries on HTTP API protocol. Supported search, insert, update, delete, replace operations. Data manipulation commands can be also bulked, also there are some limitations currently as MVA and JSON attributes can’t be used for inserts, replaces or updates.
  • RELOAD INDEXES command
  • FLUSH LOGS command
  • SHOW THREADS can show progress of optimize, rotation or flushes.
  • GROUP N BY work correctly with MVA attributes
  • blackhole agents are run on separate thread to not affect master query anymore
  • implemented reference count on indexes, to avoid stalls caused by rotations and high load
  • SHA1 hashing implemented, not exposed yet externally
  • fixes for compiling on FreeBSD, macOS and Alpine

Bugfixes

  • 989752b filter regression with block index
  • b1c3864 rename PAGE_SIZE -> ARENA_PAGE_SIZE for compatibility with musl
  • f2133cc disable googletests for cmake < 3.1.0
  • f30ec53 failed to bind socket on daemon restart
  • 0807240 fixed crash of daemon on shutdown
  • 3e3acc3 fixed show threads for system blackhole thread
  • 262c3fe Refactored config check of iconv, fixes building on FreeBSD and Darwin

Version 2.4.1 GA, 16 October 2017

Features and improvements

Compiling

Manticore Search is built using cmake and the minimum gcc version required for compiling is 4.7.2.

Folders and service

Manticore Search runs under manticore user.

Default data folder is now /var/lib/manticore/.

Default log folder is now /var/log/manticore/.

Default pid folder is now /var/run/manticore/.

Bugfixes

  • a58c619 fixed SHOW COLLATION statement that breaks java connector
  • 631cf4e fixed crashes on processing distributed indexes; added locks to distributed index hash; removed move and copy operators from agent
  • 942bec0 fixed crashes on processing distributed indexes due to parallel reconnects
  • e5c1ed2 fixed crash at crash handler on store query to daemon log
  • 4a4bda5 fixed a crash with pooled attributes in multiqueries
  • 3873bfb fixed reduced core size by prevent index pages got included into core file
  • 11e6254 fixed searchd crashes on startup when invalid agents are specified
  • 4ca6350 fixed indexer reports error in sql_query_killlist query
  • 123a9f0 fixed fold_lemmas=1 vs hit count
  • cb99164 fixed inconsistent behavior of html_strip
  • e406761 fixed optimize rt index loose new settings; fixed optimize with sync option lock leaks;
  • 86aeb82 Fixed processing erroneous multiqueries
  • 2645230 fixed result set depends on multi-query order
  • 72395d9 fixed daemon crash on multi-query with bad query
  • f353326 fixed shared to exclusive lock
  • 3754785 fixed daemon crash for query without indexes
  • 29f360e fixed dead lock of daemon

Version 2.3.3, 06 July 2017

  • Manticore branding