To shed light on the state of the in-memory database market, we conducted a survey on the prevalent use cases for in-memory databases. Respondents included software architects, developers, enterprise executives and data scientists1. The results revealed a high demand for real-time capabilities, such as analytics and data capture, as well as a high level of interest in Spark Streaming.
Real-Time Needs for In-Memory Databases
It is no surprise that our survey results highlight real-time analytics as the top use case for in-memory databases. For years, big data was heralded as the future of technology – today, it is a reality for companies big and small. Going real-time is the next phase for big data, and people seek technologies that address real-time data needs above all else. Those who can successfully converge transactional and analytical data processing, see greater efficiency in data management and have an invaluable advantage over their competitors.
The Rise of Spark
Looking at in-memory market trends, we asked our respondents which technologies they plan to evaluate in the upcoming year. Apache Spark takes the cake, with more than 50% of respondents planning to evaluate the data processing framework in the next year.
According to our survey, Spark Streaming is the most widely adopted Spark library. This shift in popularity from Spark SQL to Spark Streaming is good news for companies like Pinterest, who utilize Spark capabilities to structure real-time data up to the last click. By combining Spark Streaming with other memory-optimized technologies, like SingleStore, Pinterest can measure user engagement and developing trends in real-time.
Challenges and Solutions
In many ways, the purpose of an in-memory database is to provide a fast, persistent solution to big data challenges. The final, open-ended, portion of our survey asked our respondents what kinds of big data challenges are currently causing them the greatest strife. Popular answers include log processing bottlenecks, subpar performance, high costs, data migration from legacy systems, and complex queries for analytics. The utility and value of a database engine depends on its ability to provide solutions these common challenges. At SingleStore, we have spent the past four years iterating to offer an in-memory solution that gives users flexibility, agility, and security, with features like:
JSON Support
As Chris Preimesberger notes in an eWeek article, the SingleStore real-time, big data analytics platform is the first to combine structured and semi-structured data in a single database with JSON analytics. This integration saves users time and dollars spent on other middleware solutions.
SingleStore Loader
The SingleStore Loader is a productivity tool that allows for more efficient and streamlined data ingest, like from Amazon Web Service (AWS) or Hadoop Distributed File System (HDFS). Data ingestion is often a cumbersome and complex. The SingleStore Loader addresses this challenge by allowing direct streaming from the datastore in just one transfer, and is capable of supporting multiple parallel input streams. The result: increased performance and minimization of repetitive operations.
Spark Connector
The SingleStore Spark Connector, which unites Spark’s in-memory data processing framework with SingleStore, maximizes operational data with highly advanced analytics. Taking advantage of Spark’s capabilities means top notch analytics and high-performance parallel throughput.
Converged Processing
HTAP, or hybrid transaction and analytical processing, enables SingleStore customers to merge transactions and analytics into a single database system. HTAP facilitates analyzing large volumes of data without the need for separate data marts or warehouses. Because transactions and analytics are combined in a single database, users are able to aggregate and report on real-time data in a more advanced manner than with traditional architecture
To download SingleStore Community Edition for free, visit www.singlestore.com/cloud-trial/.
- Market Outlook for In-Memory Databaeses – 67 survey respondents
