This post recaps a session at the SingleStore NOW 2024 Conference, held at the Chase Center on October 3, 2024. To view the entire session, check out the video at the bottom of the blog.
Overview
How do you scale your data infrastructure to handle some of the most extensive identity datasets in the world? Kannan Dorairaj, Chief Architect at LiveRamp, shared how his company has done just that. Focusing on leveraging SingleStore for real-time processing, Kannan detailed the challenges and solutions involved in building a scalable identity graph that supports data collaboration at petabyte scale.
About the speaker
Kannan Dorairaj is the Chief Architect at LiveRamp, a leading identity provider that connects customer data across various touchpoints for advertisers and publishers. With over 30 years of industry experience, Kannan has played a crucial role in transforming LiveRamp’s data processing capabilities, enabling them to handle vast volumes of data while reducing costs and improving performance.
Key takeaways
Kanan’s presentation provided insights into the technical and strategic choices behind scaling data infrastructure with SingleStore.
1. The challenge of scale. LiveRamp’s journey began with the need to move from batch processing of aggregated data to handling event data at scale. As the company’s data needs grew to nearly 100 petabytes, traditional big data solutions — which relied on batch processes and multiple systems — could no longer meet the new SLAs and real-time processing requirements. LiveRamp needed to find a solution that could handle vast amounts of data more efficiently and at a lower cost.
2. Selecting SingleStore for scale and performance. After evaluating various databases, LiveRamp chose SingleStore for our ability to support both high-speed transactional and analytical workloads. Kannan highlighted a few reasons for this choice:
- Object store integration. SingleStore’s support for object stores like Amazon S3 and Google Cloud Storage allowed LiveRamp to separate storage and compute, achieving unlimited storage capacity while maintaining fast processing.
- Efficient data loading and processing. The architecture enabled LiveRamp to process large datasets, like joining tables with 50 billion records, in seconds. This performance drastically reduced processing times compared to previous setups.
- Scalability with ephemeral clusters. LiveRamp implemented domain-based data partitioning, using separate clusters for each customer or data segment. That approach allowed them to scale efficiently by handling data in smaller, manageable chunks.
3. Enabling new use cases through data collaboration. As LiveRamp shifted toward providing a complete data collaboration platform, they needed a solution that could integrate seamlessly with various data sources and support real-time event data analysis. SingleStore’s ability to quickly ingest and process data enabled new use cases, like analyzing ad campaign effectiveness in real time and providing faster insights to customers.
4. Optimizing costs and reducing your carbon footprint. Kannan shared how the shift to SingleStore significantly lowered LiveRamp’s cloud costs by eliminating the need for multiple data copies and reducing the footprint of their data infrastructure. The move also substantially reduced their carbon footprint, aligning with the company’s commitment to sustainability.
5. Simplifying data architecture. By consolidating data operations within SingleStore, LiveRamp simplified complex processes that previously involved multiple systems and thousands of lines of code. Kannan noted that tasks like joining datasets and performing large-scale queries could now be done with simple SQL statements, improving developer productivity and reducing technical debt.
Take it to the next level
Ready to unlock the power of real-time data at scale? Learn more from the complete session and start your free SingleStore Helios® trial today. Discover how SingleStore can transform your data processing capabilities, enabling you to handle even the most extensive datasets quickly and efficiently.
