Existing financial data offerings are not built for modern technology. They can’t keep up with the scale and speed needed to support innovative financial applications. Smaller developers are priced out of the market due to solutions that are too expensive, restrictive, and complicated.
The financial industry has long lagged behind other markets when it comes to technology modernization. Pre-pandemic, only 55 percent of financial services organizations took steps to implement their digital transformation plans. The pandemic helped accelerate the process for many companies, but they still had to contend with legacy systems and inefficient operations.
IEX Cloud, an IEX Group company, tackled this problem with an innovative financial data delivery platform that makes quality financial data and services accessible to everyone.
The parent company, IEX Group, was founded in 2012 with a simple mission: build fairer markets. It started by creating a new stock exchange, called the Investors Exchange, focusing on transparency and access to data to level the playing field for long-term investors, such as mutual and pension funds, that represent the savings of millions of people.
IEX Cloud was founded in January 2019. Josh Blackburn, Head of Technology at IEX Cloud, said, “IEX Cloudʼs vision is to create a vibrant data ecosystem that delivers best in class data to the individual developers and the investor community, as well as those at Fintechs and enterprises large and small. The industry has long been held back by barriers that limit access to the data they need to innovate.”
Pioneering a Better Way to Build and Scale Financial Applications
IEX Cloud’s priorities were to:
- Break down these barriers and constantly reinvent to better serve the entire ecosystem.
- Challenge the status quo in the financial data industry by delivering high-quality data to their community that is easy to use and available at a reasonable price.
- Provide a powerful set of capabilities to help developers build efficiently and iterate quickly.
- Simplify legal, modernize access, and enable flexible pricing.
The platform provided powerful functionality for developers, with a flexible, scalable pricing model based on the data used. Customers are not locked into long-term contracts and substantial upfront payments. Instead, they get open access to a range of institutional-grade datasets with one subscription, split between core and premium data.
Core data includes:
- Real-time and historical stock prices
- Corporate actions
- Full U.S. market coverage
- Cryptocurrency
- Forex
- Commodities
- Economic data
- Reference data
- IEX Investors Exchange data
- 95,000+ securities
- 20+ exchanges
- 100+ currency pairs
- And much more
Premium data comes from a growing community of curated partners, such as:
- Wall Street Horizon
- Fraud Factors
- Audit Analytics
- ValuEngine
- Stocktwits
- And much more
IEX Cloud was designed to be developer-friendly, making it as easy as possible to integrate data into their projects.
- All data is accessed through a single API. This API is based on REST, has resource-oriented URLs, returns JSON-encoded responses, and returns standard HTTP response codes.
- Support for C++, Go, .NET, R, Node.js, Python, Java, PHP, and Ruby.
- On-demand and streaming data availability.
- A no-code Rules Engine that automates serverless data analysis through custom event-driven alerts and data-driven conditions.
- Unlimited sandbox testing gives developers an environment to try free API calls and experiment with this financial data.
- The included cloud-based caching offers a fully managed service that automates data storage, scaling, high availability, monitoring, and administration.
- IEX Cloud provides a platform that can act as a developer’s backend, as some startups lack scalable infrastructure.
- Automated signup flow so users can get started immediately.
Data creators can use IEX Cloud as a turn-key distribution platform to distribute and commercialize their data. The platform takes care of the complete data delivery process by:
- Managing and scaling the API
- Tracking usage
- Billing customers
- Handling all other overhead
- Data normalization and symbology integration
These customers can integrate their data into IEX Cloud and gain instant access to a growing global user base or distribute to their pre-existing customers.
How IEX Cloud Customers Use the Platform
IEX Cloud’s flexibility and accessibility supported a wide range of use cases, such as:
- Front-office traders needing real-time data.
- Building out trade blotters and order management solutions.
- Understanding the market data and its risk in attribution implications on one’s portfolio.
- Supporting back-office operations and reconciliation.
- Keeping IR teams up to date about what’s going on in their company and near-competitors’ companies through news coverage, events, and streams of data delivered seamlessly through IEX Cloud’s API to their Business Intelligence (BI) solution of choice.
- Helping IR teams adjust legacy workflows to the work-from-home environment through a flexible platform.
Startups gain particular value from IEX Cloud. Dave Hannibal, Partnerships Manager at IEX Cloud, explained the platform’s advantage for this customer segment.
“We help them scale. They can come in and go from ideation through being a full-fledged fintech with millions of users.”
Hannibal continued, “We can work with them, and shepherd them all the way through that process, and scale with them, instantaneously. They never have to do a thing with us - as we continue to add more data on the platform, they can just grab and go. We are built in a way that helps them scale and then deliver on their value proposition. We have Fintechs all around the world who are building with us, who are integrating us, who are leveraging us for more and more data. It continues to grow across new startups and incumbents looking for new ways to redo old processes.”
Finding the Right Database to Help IEX Cloud Scale Beyond the Startup Phase
IEX Cloud had ambitious goals for its platform, but it needed to solve many technical challenges before it could offer the necessary speed and scale for modern financial applications. The company also had only three months to get the beta up and running.
Blackburn’s goal for the platform’s database was, “I wanted everything consolidated into one store, not a bunch of databases all over the place. Typically, you would split your data into different databases.”
He continued, “Amazon Redshift or Snowflake are great for storing large amounts of historical data and doing analytics. But neither database is going to meet a real-time service level agreement (SLA) for web developers, websites, or financial data. We want a delivery platform that scales to any number of users with real-time use cases, and any amount of data under the hood.”
IEX Cloud started out lean to get up and running with a few data sets. A managed MySQL database was selected initially because it was widely supported, quick to get running, and cost-effective for an early-stage startup.
The platform quickly outgrew that and ran into performance issues. As the team started adding more data and moving past the start-up phase, they needed something more web-scale.
Free Databases Are Not Free
The team chose Clickhouse to alleviate the immediate pain. It was a free solution that had high write speeds for overnight ETL processes.
However, free is not free. Clickhouse optimization was consuming a large percentage of the Site Reliability Engineers’ (SRE) time. Clustering didn’t work as advertised, and would fall over with too many concurrent reads. It was built for large analytical queries and few users.
IEX Cloud’s user base was growing, and Clickhouse wasn’t able to support millions of small reads. While it was great for time series data speed, it was too rigid. At this point, the team had data stored in MySQL, Clickhouse, and Redis.
Searching for Their Permanent Database Technology
After IEX Cloud’s first year and with revenue in the door, it was time to look for a database solution that would meet their needs and scale with the business. The team didn’t want to migrate databases again.
Financial data is complex, comes in many forms, and can change after the write. The database technology for IEX Cloud needed to be up for the challenge.
“We're trying to standardize any data set in the world, into one format and one API signature for our users. It's easier to learn. As we add data, it becomes readily available in the system,” explained Blackburn.
“We want to be able to provide dynamic metadata to our users, to inform new data on the system. If we have library maintainers or developers that access the API, they do it in one standard way. All dates are the same, all request formats are the same.”
Other technical requirements for IEX Cloud included:
- Horizontally scalable to account for a dynamically changing traffic profile and volatile markets
- Massive read and write speed
- Fast bulk data loading
- Easy to manage
- Good support
- Database technology based on standards with wide community adoption and a deep talent pool
- On-demand querying for specific data points
- Server-Sent Event streaming for real-time updates
- Filtering data for the Rules Engine
- Meeting real-time SLAs
- Support for many traditional and alternative financial data sources
- Storing and serving real-time and historical data at the same time
- Data delivery at scale for any amount of data and users
- Ability to evaluate thousands of data points per second
- Cost-efficient to support the viability of the business model
- Immediate availability of the data
- Support for fast ETL operations for hundreds of data sets per day, scaling up to thousands long-term
Benefits of SingleStoreDB for Financial Organizations
IEX Cloud evaluated many databases for its impending move, including KDB, Postgres, CitusDB, BigQuery, MySQL Enterprise, Yugabyte, Vertica, Sybase, Cassandra, and Cockroach. However, each option fell short in solving for all the use cases the team wanted to offer, except for SingleStore.
SingleStore, a unified database for fast analytics on any data, anywhere, had been on IEX Group’s radar for seven years when the company was developing the Investors Exchange. The familiarity with this database technology led IEX Cloud to revisit this solution for its new platform, halfway through 2020.
SingleStoreDB is the cloud-native, unified database built for speed, scale, and agility. It's designed to scale for today's most demanding hyper-scale applications and analytical systems. With ultra-fast ingest, super low latency, high concurrency, and the ability to get started fast anywhere, it offers powerful functionality for innovative financial organizations.
The SingleStoreDB distributed database allows financial technology companies to easily capture, process, analyze, and act on data to thrive in today’s insight-driven economy. You can run this solution on any public cloud, hybrid cloud, multi-cloud, or on-premises with commodity hardware, making it easy to add capacity at a fraction of the cost of legacy providers. Getting started is simple by design, as it’s built with familiar, developer-friendly SQL.
SingleStore features that help financial services companies modernize their technology include:
- Memory-optimized database for delivering operational analytics at enterprise scale.
- Ingestion of millions of events per second with ACID transactions while simultaneously analyzing billions of rows of data in relational SQL, JSON, geospatial, and full-text search formats.
- Ad hoc data analysis.
- Workload management to automatically deliver reliable database performance while under high concurrency.
- Distributed storage to maximize resilience and performance.
- Support for petabytes of data.
- Robust parallel execution engine for read and write queries delivering ultra-fast performance.
- In-memory and on-disk tables.
- Comprehensive security for sensitive financial data without compromising database performance.
How SingleStore Solved All of IEX Cloud’s Use Cases and Powered Its Growth
“The financial markets produce a lot of real-time data. We need to store all that, and serve at the same time. In database land, that's almost impossible,” said Blackburn. Luckily, SingleStore made the impossible, possible for IEX Cloud.
SingleStore was the only database that both met all of the requirements for IEX Cloud and was cost-effective for the startup. It also supports MySQL wire protocol, which allowed the team to use all their existing code and libraries to save months of migration work.
The initial integration wasn’t instant magic, as is the case with any software. IEX Cloud had unique requirements, and SingleStore provided responsive and knowledgeable support to get this unique build up and running quickly.
The data set at the time was small, measured in terabytes, but the team’s roadmap has them exceeding 100s of billions of records daily. They wanted a foundation capable of scaling automatically and horizontally to petabytes as the platform grew.
The team also found that the database’s deployment and administration process was mature, and SingleStore was responsive in assisting them with the rollout. They chose Google Cloud Services (GCS) for the implementation due to its interface and extremely fast network. SingleStore’s native support for Kafka and GCS pipelines allowed IEX Cloud to automate its ETL processes, so all the team needed to do was point at the GCS bucket.
Blackburn went into detail about the complexity inherent in working with financial data. “The interesting thing about finance is when you’re doing analysis of a point in time. You want to know what affected the market and why something happened. When you have hundreds of datasets that could have impacted that, the power is in the more datasets you have to analyze.”
“We’re not in the business of selling storage, we’re in the business of helping deliver the data where you need it to go and having that economic model on top.”
“Looking at another platform like Snowflake or AWS, our delivery platform is a bolted-on feature. It’s not a deliberate focus on, “How do I standardize my delivery?” or “How do I work on authentication, permissions, and all the complexity and metadata in the financial space?”
“It’s not just a financial data problem. Many people have problems standardizing and delivering their data. How do you monetize or deliver that data? We need to be able to standardize all our data to make it efficient and to address that other point of making this accessible to people who don’t have a large amount of money. SingleStore solved for all those use cases.”
The Collaborative Potential of a Symbiotic SingleStore Partnership
IEX Cloud’s partnership with SingleStore goes beyond using the database technology. Hannibal discussed the collaborative and complementary nature of this relationship and the future possibilities.
“Where SingleStore provides a database, we've used it in a unique way to create a whole distribution platform that is a value-add for SingleStore’s clients, too.” Customers can use IEX Cloud for data distribution and SingleStore for their master data projects.
“You quickly see the symbiosis through what SingleStore and IEX Cloud has. That's different versus the other players in the space. So, from a business perspective, that was very interesting and appealing to me.”
What IEX Cloud Achieved With SingleStoreDB
IEX Cloud has grown significantly following the adoption of SingleStoreDB for its financial data delivery platform, resulting in:
- Over 2.5 billion API requests processed daily with an 8ms average response time.
- 200+ financial data sets.
- Internal data handling 400,00-800,000 ops per second.
- Intraday ingress of 60-100 MB/sec and egress of 120-160 MB/sec.
- After only 2 years, they gained 150,000 users in 120 countries, representing 20 million unique users downstream.
- 10-15x speed improvement over the previous database.
- More cost-effective than scaling MySQL to meet performance and functionality requirements.
- The equivalent of 2 full-time-employees’ worth of time saved out of a team of 20 when switching from MySQL to SingleStoreDB.
- ETL process execution time dropped from days to minutes.
“This system upgrade was a critical step for IEX Cloud, and not just because we expanded our immediate data offering. Behind the scenes, we've solidified partnerships with best-in-class data creators that will enable us to continue enhancing the quality, coverage, and timeliness of our data. Meanwhile, weʼre able to continue providing valuable datasets at an accessible price point. The result: an entirely new business model for financial data that meets the needs of individuals and businesses across the board,” said Blackburn.
What's Next for IEX Cloud?
IEX Cloud has an ambitious roadmap ahead of it for 2021. The ability to onboard data and get it out through the API faster expands the data and coverage available on the platform, improving the end-customer experience.
Upcoming features and financial data types include a comprehensive reference data model, global indices coverage, IPO calendar data, 147 global corporate actions event types, global futures end-of-day pricing and reference data, and historical fixed income data.
Are you ready to harness the power of the world’s fastest database for operational analytics? Contact us to discuss ways we can collaborate or get started with SingleStore DB for free.
