SingleStore Kai™: Real-Time Analytics Benchmarks

Based on extensive internal and external benchmarks and performance testing, our recently released API SingleStore Kai™ delivers 100x faster analytics over MongoDB — and returns results more than 1,000x faster for some queries.

SingleStore Kai™ also exhibits consistently better performance and resilience, with the performance gap widening as data volumes grow, or as operations scale. Overall, SingleStore Kai™ offers an easy and effective API to turbocharge analytics on MongoDB, while also delivering significantly better price-performance on analytics.

SingleStore Kai™ provides a fast, easy and powerful API to significantly accelerate MongoDB applications — without any code changes or data conversions. You can achieve real-time analytics and low-latency aggregations using familiar MongoDB client drivers, tools and APIs, and can quickly connect any application written for MongoDB to SingleStore Kai™ simply by changing the connection string.

MongoDB is a document database that isn’t designed to power fast analytics or large-scale aggregations on JSON. In contrast, SingleStoreDB supports a wide variety of workloads, and is the ideal solution to these growing pains — either as either a drop-in full replacement, or a turnkey augmentation for your primary database. Now, with SingleStore Kai™ , those options are available to applications and workloads built around MongoDB and other MongoDB-compatible databases.

For more details, check out our SingleStore Kai™ documentation.

How Does SingleStore Kai™ Work?

SingleStore Kai™ implements the MongoDB wire protocol and provides a mapping from MongoDB’s non-relational organization of JSON-like documents to SingleStoreDB’s distributed organization of relational SQL tables. To the application, SingleStore Kai™ appears to be an ordinary MongoDB-compatible database. Behind the scenes, it accelerates MongoDB operations using SingleStoreDB's highly optimized storage and execution engines.

We did extensive performance testing and benchmarking —  including both internal and external industry benchmarks — that we will cover in more detail in this blog.

However, the advantages extend beyond performance: SingleStore Kai™ also enables MongoDB applications to leverage additional functionality not available in other MongoDB-compatible databases — including vector operations and data pipelines — and to effortlessly share data with applications written using a SQL API. For a deeper dive, read through our blog introducing SingleStore Kai™. For some exciting benchmarks, read on!

Real-Time Analytics Example: Internal Benchmarks

We developed a simple benchmark anyone can use to quickly observe the benefits of SingleStore Kai™ relative to MongoDB or other compatible databases. To keep it accessible and transparent, this benchmark uses only a tiny NodeJS program for loading data and executing queries.

The program generates an infinite amount of synthetic event data from fictitious IOT devices. This represents data that might be reported by IOT devices distributed around the world and collected into a central database. While loading, the program executes the following analytical queries continuously, mimicking a real-life scenario.

Q1. What is the most recent event sent by any device?

[
  { $group: { _id: null, max: { $max: "$eventTime" } } }
]

Q2. Which five devices have sent the most events?

[
  { $group: { _id: "$deviceId", count: { $sum: 1 } } },
  { $sort: { count: -1 } },
  { $limit: 5 },
]

Q3. What is the latest status reported for each version 2 device?

[
  { $match: { deviceVersion: 2 } },
  { $sort: { eventTime: -1 } },
  { $group: { _id: "$deviceId", latest: { $first: "$status" } } },
  { $group: { _id: "$latest", count: { $sum: 1 } } },
]

Q4. What devices have not sent an event in the last hour?

[
  { $sort: { eventTime: -1 } },
  { $group: { _id: "$deviceId", eventTime: { $first: "$eventTime" } } },
  {
    $match: {
      eventTime: { $lt: new Date(new Date().getTime() - 1 * 1000 * 60) },
    },
  },
  { $sort: { eventTime: -1 } },
]

Q5. What is the average value reported for each location in an ONLINE event?

[
  { $match: { status: "ONLINE" } },
  { $group: { _id: "$deviceLocation", avg: { $avg: "$value" } } },
  { $sort: { avg: 1 } },
]

Q6. What is the total number of events with either the ADD08 or LOW07 tag?

[
  { $match: { codes: { $in: ["ADD08", "LOW07"] } } },
  { $count: "count" },
]

These are the sort of queries that an application monitoring IoT devices might need to execute in real time. And to provide a usable experience the results need to be fast and accurate.

We ran this benchmark on a MongoDB Atlas M80 NVME and a Singlestore Helios S2 instance, which have similar vCPU and RAM resources.

Internal benchmark — sizing

We let the benchmark run until it had loaded 40 million rows, which took just under nine minutes for both databases.  Then, we measured the time taken to execute each query.

Benchmark — query latency in seconds (lower is better)

At 40 million rows, Singlestore Helios S2 with SingleStore Kai™ demonstrates significant performance advantage over MongoDB Atlas M80 NVME. The slowest query was about 29x faster on SingleStoreDB with Kai compared to MongoDB, while the fastest ones saw a more than 800x improvement.

Even more fascinating is the performance comparison over time as the amount of data increases. Plotting the query times taken as the benchmark was running shows SingleStoreDB with Kai held the advantage throughout the test, even as the results it was calculating represented real-time analytics results over the data loaded so far.

As you can see in the following plots, the time taken by MongoDB dips slightly at the end. This is because at that point of the test, the data load had finished and MongoDB executed the queries a little quicker without that load. This means while the database is under a read/write workload, the multiples are even higher than displayed in the previous summary table. SingleStoreDB consistently performs faster queries — even during real-time ingestion and increasing event counts.

These results demonstrate that:

1. SingleStoreDB performs orders of magnitude (20-800x) faster for analytical queries compared to MongoDB at any point in time
2. The Singlestore Helios S2 scales more effectively than the MongoDB Atlas M80 NVME instance with the performance gap widening as the amount of data increases.
3. SingleStore Kai™ enables large-scale analytics and aggregations at a performance level previously inaccessible to MongoDB applications.

MongoDB Indexes Don’t Help Either

The default benchmark runs without any indexes. To give the MongoDB Atlas M80 NVME instance another chance and relative advantage, we ran it again with indexes on every field. This micro-management of indexes is a step not typically required with SingleStore Kai™. Nor did it help — load times and performance were both worse than the default time-series collection results previously shown, so we did not include the results here.

What About Operational Benchmarks?

While the preceding numbers demonstrate scaling of analytical queries, SingleStore Kai™ (with MongoDB® compatibility) demonstrates good results for transactional workloads as well. Take, for example, the following operations from the previous collection:

We took the median of 10 trials.

In this particular case, deleting 100 documents by _id and inserting the same one at time was faster on the Singlestore Helios S2 than the MongoDB NVME. This is just a small illustration of operational potential and is not the focus of this blog post, but in many cases SingleStore can outperform on both transactional and analytical workloads.

Real-Time Analytics Example: External Benchmarks

The results from our small internal benchmark illustrate some of what applications can achieve with SingleStore Kai™. We did extensive testing using other industry benchmarks as well, and found external benchmarks further strengthen and corroborate this outcome.

One of these benchmarks we tried with SingleStore Kai™ is the MongoDB version of the ClickBench analytics benchmark. This public, reproducible benchmark showcases the performance of analytical queries on a data set just shy of  100 million documents. We compared the performance of this benchmark on the same SingleStoreDB S2 and MongoDB Atlas M80 NVME instances described in the previous section. The results show that workloads with this sort of analytical query can benefit from SingleStore Kai™.

Key findings:

• More than 80% of the queries completed in less than half the time
• More than 50% of the queries are at least 43x faster
• More than 30% of the queries are at least 100x faster
• Only two queries are (very slightly) slower and are sub-300ms

Here is a full summary of the benchmark results:

Benchmark — median query latency in seconds (lower is better)

Again, as evidenced from the results in the table, SingleStore Kai™ demonstrates a significant performance advantage over MongoDB Atlas M80 NVME. The slowest query was about the same as MongoDB, while the fastest ones were more than 100,000x faster. Eliminating the outliers, 80% of the queries were completed in less than half the time, compared to MongoDB.

One quick note — three of the 43 queries are excluded from these results because at the time of publication, they are not yet supported by SingleStore Kai™.

In terms of compatibility, SingleStoreDB with Kai provides compatibility and coverage that is higher than other leading Mongo API-compatible databases — including Azure CosmosDB and AWS DocumentDB — for both transactional and analytical commands. To learn more about the detailed compatibility and mapping for SingleStore Kai™, head over to our documentation page.

To close out the analysis on the ClickBench performance, take a look at  the following scatter plot of the previous results comparing performance across SingleStore Kai™ and MongoDB across all of the queries to show the relative spread. As evidenced, SingleStoreDB with Kai delivers consistent and improved query performance for queries compared to MongoDB — which demonstrates wide variations and extensive gaps in query performance.

The speedup is easy to appreciate. It’s apparent that these types of analytical queries cannot be performed interactively over MongoDB. With SingleStore Kai™, they complete quickly enough over SingleStoreDB to power real-time decision making.

A Final Word on Price Performance

For the tests we covered, the MongoDB Atlas M80 NVME we utilized cost $7.85 per hour. For the long query times of the MongoDB Atlas M80 NVME (and performing three iterations per query) this leads to an expensive benchmark! The MongoDB portion of the benchmark cost about $100.

In contrast, the Singlestore Helios S2 with SingleStore Kai™ cost just $5.20 per hour and required significantly less time to execute the benchmark queries. Overall the time required for the SingleStoreDB S2 is less than $10 charge, about one-tenth the cost for comparable performance for MongoDB.

It’s important to note this was done on a very limited dataset and scope, simply to execute the benchmarks. But, as a developer or a decision maker, you can easily extrapolate the impact this can have on your costs and bottom line when scaling your data (to hundreds of gigabytes or terabytes) to power your real-time applications and workloads. This lower price with higher performance makes SingleStore Kai™ a cost-effective choice for workloads, like the ones covered here.

For more information:

• Learn more about SingleStore Kai™
• Try SingleStore Kai™ for free
• Schedule a deep dive with one of our Solution Engineers
• Watch the SingleStore Kai™ demo.
• Dig deeper into the SingleStore Kai™ documentation
• Explore the benchmark Github