Leading Institutional Crypto Data Provider DAR Lowers TCO 50% and Expands Client Base 600% with SingleStore

Cryptocurrency offers a wealth of opportunities for institutional investors, but getting clean and accurate market data to make the right trading decisions can be a challenge for institutional investors. Market manipulation, biased solutions, and lack of standardization in this sector have kept many institutional participants on the digital asset sidelines. 

Digital Asset Research (DAR) wanted to change this landscape for asset managers, banks, custodians, family offices, fintech firms, fund administrators, hedge funds, and venture capital firms by providing ‘clean’, objective pricing and verified volume data for 3,100+ digital assets. 

Since 2017, DAR leads by rigorously vetting out noisy inputs for flagship clients such as Bloomberg, FTSE Russell, and Wilshire. Launched in 2017, the company brought expertise in traditional finance and the digital asset space to deliver a range of product solutions to navigate the cryptoverse:

• Clean intraday and close pricing data

• Indicators such as market cap, free float, and volume statistics

• Qualitative and quantitative data on digital asset exchanges

• Token data including data on technology practices, security practices, and developer activity

• Market structure data, including price discovery analysis

• Comprehensive calendar information about events affecting the digital asset ecosystem

Challenges/Goals 

Separating the signal from the noise in digital asset pricing is a complex process. In order for DAR’s platform to do this, it needs to be able to vet assets, analyze trades, and produce a rich data set to tie it all together in a fast-paced, 24x7 market. 

The platform it would build needed to handle both real-time and historical data analysis, with 60 billion historical trades and two billion historical aggregated prices available alongside the latest market data. Some digital asset data sets, such as Bitcoin, have more than a decade of data. 

Performing network analytics of all transactions on a chain could involve tens of billions of transactions to create the one-second price and everything in between for an asset. The Digital Asset Research platform collects data from 450+ exchanges in duplicate so it can run multiple processes that are not reliant on each other.

Along with pricing data, DAR also aggregates data for a Crypto Events Calendar to provide a centralized view of events, technical developments, policy changes, token and product launches and listings, and other important news in the crypto world. 

“There’s a lot of data and queries coming into the platform at the same time. With our previous database, it was hard to keep track of it in a consistent way,” explained Michael Zimberg, Chief Technology Officer, Digital Asset Research.

Michael Zimberg, Chief Technology Officer, Digital Asset Research

Powering this type of solution requires a massive amount of data. Here’s a look at DAR by the numbers:

• 3,100+ crypto assets priced 
• 140 million trades per day
• 450+ exchanges evaluated
• 450+ SEDOLs assigned to digital assets

Technology Requirements 

The technology requirements to support such a data-intensive platform included: 

• A performant analytics engine with columnstore to democratize data access 
• Real-time analytics for very fast reads and quick ingestion
• A single, unified data store 
• Low total cost of ownership (TCO), as DAR is a startup with a lean engineering team 
• Scalability to support its accelerated growth trajectory in the digital asset market

“When we were running the platform on AWS Aurora (MySQL), it had a hard time deduplicating and allowing other users to query data. If there were six pricing engines running against it as it was updating, there were all kinds of exclusive locks. The database would get backed up and go down every few days,” lamented Zimberg. 

Aurora also had limited analytical capabilities and was cost-prohibitive to run at the scale DAR required. A new database solution was needed to get DAR to the speed and scale it needed to achieve its aggressive goals.

Why SingleStore 

“I knew pretty quickly that AWS Aurora wasn’t going to be a great solution for us, so we started experimenting with Snowflake and SingleStore at the same time. We used a firm called Data Sleek to get our MySQL more stable and help us come up with a better environment,” said Zimberg. 

Price performance became the key factor when choosing between the two technologies. 

“Unfortunately with Snowflake, their pricing transparency just wasn’t there. We got a big surprise the first month we went to use it, so that became a nonstarter,” Zimberg explained.

While Snowflake had better query speeds than Aurora and a high compression rate, it didn’t deliver the exceptional performance DAR was looking for. SingleStore stood out from the start with its cost-effectiveness, and brought the necessary high performance capabilities along with it. 

“When we saw that our costs would actually go down with SingleStore, it was an easy decision to migrate everything,” said Zimberg.

SingleStore is the World’s fastest cloud database for the data-intensive era.™ SingleStore features and benefits that stood out the most to DAR included: 

• Columnstore capabilities with high compression rates for DARs massive data sets 
• Streaming data and analytics that dramatically minimize time to insights with real-time streaming ingest and blazing fast queries
• Effortless scaling of applications and data sources with a tiered-storage model that offers cloud elasticity and infinite scalability
• A cloud-native managed service offering with AWS support 
• High concurrency
• A robust, user-friendly pipelines tool for setting up data ingestion in minutes
• Unbeatable price performance 

“When we were running on AWS Aurora, I couldn’t check for things that I was worried about as the CTO, such as when we last got data from certain exchanges and what the trends were,” said Zimberg. “In SingleStore, I can actually run queries in real time.”

Solution 

DAR opted to deploy SingleStore Helios. DAR had 120 AWS EC2 instances and a platform built around MySQL. With SingleStore, for most of these instances, all Zimberg and his team had to do was change the connection string to get it working. The deployment took only three weeks, as DAR did a lift-and-shift to start and brought historical data over later. In only a month, DAR was completely migrated off AWS Aurora to SingleStore.

“The ability to leverage SingleStore as a managed service was very helpful to us. Unlike a lot of the other markets, the crypto space is 24x7. We can’t have any downtime. Our real-time prices need to run all the time and that was challenging to do with AWS Aurora. SingleStore Helios met all our requirements,” said Zimberg.  

During the process of optimizing performance on SingleStore, DAR moved to a message-based architecture for processes that need the highest performance, and for that it uses Confluent Cloud and AWS. DAR also uses shard keys and columnstore to further drive performance improvements.

Outcomes

With SingleStore, DAR has massively expanded its customer base, cut its TCO in half, and improved productivity.  “The really big business outcome is that our platform can do everything we need it to in less than a second,” said Zimberg.

1,000X Faster Data Ingest = Real-Time Data for DAR Clients

The 1,000X faster data ingestion it gained with SingleStore provides real-time data for DAR’s clients and speeds time to market for DAR itself.

“We found that SingleStore pipelines are especially useful. With them, we can pick up something new that we haven’t consumed before. Even though I’m the manager, I actually write the pipelines myself because they’re pretty straightforward to use. In some cases, within minutes I can load it into production,” elaborated Zimberg.

“SingleStore handles this extremely well. We’re able to consume data in duplicate, deduplicate it, normalize it, load it into the database, and feed it to our pricing engine in under one second.“ 

Modern Architecture Enables a 50% Reduction in TCO

SingleStore’s modern architecture, AWS support, and price-performance capabilities decreased DAR’s TCO by 50% compared to AWS Aurora MySQL and Snowflake. 

Processing 4-7X More Data — with Extreme Performance 

DAR scaled from 20-30 million daily orders to 140 million after switching to SingleStore, with plenty of room to grow. The performance was also a breath of fresh air for the company. 

“Some batches are running in about seven milliseconds. That’s nothing. They’re so unbelievably fast,” said Zimberg. “In AWS Aurora, there would be an operation on the database that would take an exclusive lock that took minutes. SingleStore enables us to be in the order of thousands of times faster.”

1-Second Price Refresh Helped Make DAR the Digital Asset Pricing Provider on the Bloomberg Terminal 

“We had pricing that refreshed every 15 seconds on our old infrastructure,” said Zimberg. “Now, with SingleStore, our pricing refreshes every second quite easily. This has enabled DAR to become the pricing provider on the Bloomberg Terminal.”

SingleStore also enabled DAR to start consuming order books to provide the best bid and ask on the prices provided. It took DAR just 10 minutes to ramp up with SingleStore’s pipelines to model the table, look at the JSON object, and begin consuming the batches for this process.

Expanded its Client Base by 600%

SingleStore allowed DAR to easily add features here and there to better cater to its institutional clients. It grew from four clients a year ago to more than 25 clients today, representing more than 600% growth.

Improved IT and Workforce Productivity

 “Not worrying about patchwork and focusing on building a better platform is a life changer,” said Zimberg.  With SingleStore, both he and the team no longer have to worry about emergency outages at one o’clock in the morning or spending 10 hours to recover 30 minutes of lost real-time data.

---

To hear Digital Asset Research discuss its innovative implementation of SingleStore, watch this on demand webinar: Accelerating Fintech Applications at Scale.

SingleStore is helping companies compete and win across every vertical. Learn more →