We are thrilled to tell you that SingleStore 8.7 is here! A summary of the release is as follows:
Full-text Search Enhancements
A new Java Lucene-based full-text search implementation that supports search over JSON as well as text, fuzzy search, proximity search, boosting and BM25 scoring. These are powerful features for hybrid search and GenAI, in conjunction with vector search. The new full-text search engine is the same one used in ElasticSearch and Solr.
Vector Search Enhancements
- Improved index-based ANN search to now include index-based range search. This can use an ANN index to find all records with a vector similarity score above a threshold, not just find the top K matches.
- Updated the FAISS vector index library to the latest version, 1.8.0, from 1.7.4.
- Made the default VECTOR type output format be binary, improving performance
SQL Improvements
Added support for scalar subselect with correlating conditions, one of the commonly requested SQL extensions in Forum & support cases
Storage Improvements
- Added a command to explicitly warm the disk cache with columnar blobs for specific tables. You now have the ability to more easily proactively warm specific columns or the entire table. Previously, we only supported warming indexes.
- With this release, you can experience up to 2X query speedup when fetching data blobs from the object store for bottomless databases. All of this is achieved transparently in the background without any special input needed from the user.
Monitoring and Observability Improvements
- Added the ability to observe workloads by workspaces
- Historical Workload Monitoring now provides additional visibility on bottomless & disk spilling operations, it will help you understand,
- Disk Spilling Bytes - Amount of data spilled onto disk per execution
- Blob Cache Miss Bytes - Amount of data downloaded from bottomless to disk per execution
- Blob Cache wait time - Amount of time a query had to wait for the bottomless data to be downloaded to disk
- Instantly receive alerts for any pipeline errors to ensure smooth and uninterrupted operations
Performance Improvements
- 8.7 is the first release to support Projections (supporting secondary short and shard keys for tables to improve performance for GROUP BY, DISTINCT, range queries, key lookups and more) from the beginning. Projections were first delivered in 8.5.14 on April 2, 2024.
- Added ability to obtain histogram statistics on JSON & BSON keys, this will help the optimizer to find the best overall plan when JSON or BSON keys are used in the query.
- Introduced a new tool to help with query tuning “Feedback reoptimization." This tool will help fix bad plans that are caused by poor estimates.
Ingest and Egress
- Native Ingest of Iceberg Tables using PIPELINEs
- Change Data Capture (CDC) Out – public preview; enables feeding downstream systems the contents of application updates run on SingleStore.
- parallelized CDC from MySQL and MongoDB
- Schema inference for pipelines – allows easy file loading with little or no coding
Security
Customer-managed encryption keys (CMEK) for Helios – allows customers to revoke all access to their data if desired, including from cloud service administrators.
Kai
- Multi-key indexes are now supported on Kai to index nested arrays of values. This leverages a standard SingleStore index on a multi-key table.
- Kai now supports vector similarity search with the $vectorSearch option.
Multi-Tenancy Improvements
- As of 8.5.20, May 16, 2024, SingleStore fully supports eviction of idle tables’ code modules and skip lists from RAM. This can allow you to have ~3x as many tables depending on application characteristics.
- In 8.7.1, the “full” mode for idle table eviction is now the default, making the feature available with no effort on the part of the developer.
Please check the 8.7 release notes and documentation to learn more!
SingleStore Product Management Team
SingleStore, Inc.