Across industries, nearly every business is in the data business—which creates enormous dependencies on popular data warehouses like Teradata, Snowflake, Google BigQuery and Amazon RedShift.
Unfortunately, whether built with ported-over legacy technology or born in the cloud, these data warehouses typically depend on rigid, batch-oriented ETL/ELT technologies to capture, ingest, cleanse and transform data before it is available for analysis and reporting.
Sure, these traditional workflows can strong-arm data into a structured format that fits a predefined schema. But they are poorly suited for the data-intensive applications that form the heart and soul of today’s enterprises. Specifically, traditional data warehouses suffer from four bottlenecks:
1 – Limited support for streaming ingest: Data warehouses were not architected for parallel, high-throughput ingestion of streaming, real-time data.
2 - ETL batch windows: Batch windows inject significant delays into the data flow; dashboards and reports reflect past events rather than present conditions, displaying data that is hours or days old.
3 - Query latencies: Data warehouses were not optimized to handle the low-latency queries required for fast analytics applications and interactive dashboards.
4 – Concurrency limitations: Traditional data warehouses break down under high concurrency workloads supporting large groups of users and can be expensive to scale.
Blowing the walls out
To all companies that depend on data warehouses, SingleStore poses a bold opportunity: What if you could achieve 100x performance improvements over traditional data warehouses and associated data pipelines, running data-intensive analytic workloads, and achieving significant cost reductions? Let’s take a look at three companies that are using SingleStore for data warehouse augmentation (DWA) to achieve improvements of this dramatic magnitude—literally blowing the walls out of their data warehouse constraints.
Customer #1: Leading mobile phone manufacturer delivers real-time data visibility to executives
Situation: Senior executives at this fast-moving electronics manufacturer rely on a Tableau dashboard to monitor the real-time sales and market movements of mobile devices, which requires visualizing data by device, region, price point, product attribute and many other dimensions. Teradata, which powered this executive dashboard, couldn't scale to handle the data growth and concurrency requirements of 400+ queries per second.
Challenge: Slow and lagging performance of the executive dashboards meant users had to wait many hours to obtain fresh insights. These delays adversely impacted product launches, marketing campaigns and supply chain operations. For example, managers could not quickly determine how much raw materials were required to satisfy fluctuating consumer demands. Additionally, the company had to ingest 4 billion rows of new data each day, leading to delays of up to 10 hours to process and display the latest data in the dashboard.
SingleStore Results: Augmenting Teradata with SingleStore enabled the electronics manufacturer to transform day-old analytics into real-time insights, boosting data ingestion rates to 12 million rows per second. SingleStore significantly improved performance, delivering query responses in less than 100ms, even with high concurrencies of more than 160,000 queries per second. Finally, SingleStore’s native connection to Tableau made it easy to populate fastboards (real-time dashboards) via the MySQL wire protocol, enabling a direct Tableau-to-SingleStore interface.
Customer #2: Real-time threat analytics catapult a leading cybersecurity organization over competitors
Situation: When it comes to monitoring and reporting on potential security breaches, malware attacks and other threats to network security, every millisecond counts. This cybersecurity organization depended on Snowflake to power threat analytics and incident reporting.
Challenge: There was significant lag time—up to five minutes—between potential threat detection and reporting, eroding the firm’s competitive position. This latency was driven by a combination of factors, including difficulty with streaming data ingestion and supporting a growing volume of queries. With loads upwards of 1,000 concurrent queries per second, Snowflake just couldn’t keep up.
SingleStore Results: Since augmenting Snowflake with SingleStore, the cybersecurity firm has dramatically reduced the time it takes to detect and report emerging threats. Customers now receive threat-detection alerts and reporting in less than one second. Compared to approximately three minutes previously, this 180x improvement vastly improves the customer experience. SingleStore reduced data-ingestion latency by 15x for millions of records, achieving real-time streaming ingestion from Amazon S3, to fuel query response times of less than 500ms latency even with thousands of users concurrently accessing the application.
Customer #3: Media company boosts ad sales with fastboards
Situation: More than 100 sales reps at this large North American media conglomerate depend on a Looker dashboard to understand ad inventory and performance in order to sell ad slots to customers. Unfortunately, the Amazon RedShift data warehouse that powered the dashboard was too slow to process transactions and display results, leading to hours-long delays between when ads were sold and when the sale was reflected in the dashboard.
Problem: Ingesting new data from Amazon S3 into Redshift took an average of two hours. Furthermore, with hundreds of sales reps accessing the same dashboard concurrently, more than five minutes passed before queries were returned when the dashboard was filtered or refreshed. Account executives inadvertently found themselves closing deals for ad spots that had already been sold by their colleagues. With digital ads accounting for 32% of the medica company’s total revenue, this problem was not only damaging customer relationships, but hitting the bottom line.
SingleStore Results: Augmenting RedShift with SingleStore enabled the media company to transform its dashboard into a fastboard for sales reps, continuously ingesting new records from S3 in less than two seconds. Query response times improved by a factor of 300x; more than 1,000 ad execs can refresh their dashboards concurrently in less than one second, boosting ad sales and effectively eliminating double-booked ad spots.
The value of data warehouse augmentation
Is your organization stymied by an outdated data warehouse architecture? Not sure? Ask yourself these questions:
- Do you struggle with stale or slow-running dashboards or applications that don’t reflect the freshest information?
- How about customer experience issues, performance problems, or escalating costs with your data warehouse environments?
- Do you need to break down the barriers of slow batch processes or accelerate time-to-insights?
- Are you trying to move toward real-time or near-real-time insights or use cases?
- As you scale analytic systems to keep up with escalating data volumes and rising customer demands, do you spend excessively to upgrade hardware and software infrastructure, or pay exorbitant usage charges to cloud providers?
- Do you face a backlash from impatient users who are unable to seize data-driven opportunities?
If the answer to any of these questions is ‘yes,’ it’s time to consider blowing the walls out of your data warehouse with SingleStore.
\ To learn more about DWA architectures and how customers are using SingleStore, read our new eBook “20 - 100x Faster Analytics Through Data Warehouse Augmentation: Bring Critical Analytic Workloads into the Modern Age.” Or get started with SingleStore today and take advantage of our offer for \$600 in free product credits. Follow us on Twitter @SingleStoreDB and in our new Twitter channel for developers: @SingleStoreDevs.
