Why Fathom Analytics Ditched MySQL, Redis, and DynamoDB to Power the World's Fastest Website Analytics Platform

Challenges/Goals

Fathom Analytics is a rapidly growing website analytics software company focused on providing a simple, lightweight, privacy-first alternative to Google Analytics. Its user-friendly, privacy-centric approach is wildly popular with users, resulting in rapid growth and the need to effectively scale.

Co-Founder Jack Ellis explained, “For over a year, we'd been struggling to keep up with our analytics data growth. Fathom had been growing at the speed of light and our data ingestion had been going through the roof. Our popularity has been great for business and user privacy, but it wasn't good for application performance.”

Fathom Analytics started out using MySQL with Amazon RDS as the database to power its application, but the team encountered several challenges keeping up with its IoT analytics data growth, including: 

• Cost-prohibitive deployment with Amazon RDS for MySQL

• Poor scalability, as MySQL was struggling to handle table aggregations at a medium scale (despite the fact they were pre-aggregating data)

• Poor performance with dashboard requests getting timed out

• Connection limits

• Limits on IOPS, which meant they had to overspend on provisioning IOPS with RDS, and had no ability to scale those IOPS down with ease, leading to ever-increasing cost of ownership 

• Lack of support for data drill-downs and filtering 

  Fathom’s goals included: 

• The ability to remain independent, self-funded, and sustainable so they would not have to compromise data privacy by selling advertising and user data. 

• Meeting the needs of thousands of users with billions of page views across all organization sizes, from small businesses to Fortune 500 companies. 

• Improving the user experience, performance, and scalability of the application.

Technology Requirements

The database technology capable of supporting Fathom’s website analytics application needed to meet several important criteria: 

• Massive concurrency: Fathom’s rapidly growing user base required thousands of concurrent reads and writes without running into issues
• Manageable cost: The total cost of ownership (TCO) for the database needed to stay under $5,000 per month
• Fast time-to-value: The ideal database would offer ultra-fast, real-time analytics and high-performance queries to help users see their website metrics faster
• No roll-ups: The database solution needed to support aggregations across billions of rows and respond in milliseconds, not minutes
• High adoption by the enterprise: Fathom wanted to work with mature software that had customers much larger than itself using it
• Customer success: The database provider would deliver great customer support and well-written, easy-to-understand documentation
• Fully managed database: Finally, Fathom wanted a fully managed database-as-a-service. Ellis explained: “We're not database experts and would rather have true professionals manage something as important as our customers' analytics data.”

Why SingleStore 

Fathom reviewed other databases including Elasticsearch, TimescaleDB, InfluxDB, Rockset, ClickHouse, and Amazon Athena, and ultimately chose SingleStore Helios to support its growing website analytics application. SingleStore enabled fast ingestion, high performance queries, and real-time analytics for Fathom through a fully managed, unified database-as-a-service supporting millions of real-time queries across tens of thousands of users. The benefits included ultra-fast data ingest, MySQL wire protocol compatibility, sub-second query latencies, one-click deployment, and a petabyte-scale cloud-native data platform. 

Ellis spoke about why SingleStore was Fathom’s database of choice: “I want the whole package. I like speed, but I also want to feel good about what I'm using. I want the people we're working with to be good people. And the technology has to fit into my existing expertise so the learning curve isn't too large.”

Jack Ellis, Co-Founder, Fathom Analytics

Solution

Fathom Analytics was previously using:

• RDS MySQL for its core application database and analytics
• Redis for its application cache
• DynamoDB for high-scale ingest hash tables

Now it is using SingleStore for everything.

One of the key selling points for Fathom was that it didn’t have to rewrite any queries — thanks to SingleStore's MySQL wire protocol compatibility — which saved the team weeks of work.  

Fathom was also able to replace Amazon Elasticache (Redis) as its application cache and is using SingleStore for all of its caching needs, thereby simplifying its architecture and driving down costs significantly.

Six months after migrating its analytics database, Fathom also migrated from Amazon DynamoDB to SingleStore for ingesting hash tables, which Fathom uses to track whether users are unique. The team previously was nervous about using a non-serverless database for this but, after seeing how quickly and easily it could expand the cluster (without incurring any downtime), it was confident this could work for its operational needs. Just as importantly, Fathom ended up saving $3,000/month just by migrating away from DynamoDB.

"We are now all in on SingleStore Helios, which has allowed us to drop Redis, DynamoDB, and MySQL, saving us an absolute fortune in monthly costs, while dramatically improving performance," said Ellis.

“In March 2021, we moved all of our analytics data to the database of our dreams," said Ellis. "With SingleStore, we can perform tens of thousands (probably more at times) of inserts per second, along with utilizing it for all key/value lookups (for which we previously used DynamoDB). It handles all of our application rate-limiting and supports our unique tracking methodology, all at a fraction of the cost — what a time to be alive.”

Outcomes 

Fathom Analytics originally used a dedicated data-export environment in AWS Lambda, which regularly ran for many hours, costing Fathom a small fortune. In addition to that, the team frequently encountered data export errors for the biggest customers, resulting in hundreds of automatic retries followed by hours of work preparing manual data exports. Since moving to SingleStore, data exports to AWS S3 are achieved via a single query which finishes in less than 30 seconds. The Fathom team can now spend that time on improving the platform. 

Freed Up Significant Mental Bandwidth

Its new simplified infrastructure freed up significant mental bandwidth within the company. Fathom is now able to focus on developing its product rather than worrying about its data. 

Reduced TCO by 60+% and Simplified Infrastructure

Fathom Analytics wanted to keep its database costs under $5,000/month, and SingleStore Helios delivered cost-efficient top performance at only $2,500 per month. The team also reduced its AWS bill by more than $3,500 per month, as it no longer needed Amazon RDS for MySQL, an expensive NAT gateway; Amazon Elasticache (Redis) and DynamoDB for caching; or expensive over-provisioned storage for RDS IOPS. Fathom's CPU overhead dropped by 60-80%, and migrating to SingleStore was an incredibly smooth process.

“We’re now in a position where we have a single database for our entire application. Highly available, infinitely scalable, with great value. What excites me the most is that any time we have to spend more money on SingleStore and add CPUs and RAM, we know our dashboards will perform faster. We’ve put ourselves in a great position here. Many developers will be reading this who are managing Redis, DynamoDB, MySQL, and whatever else they’ve got in their stack. It’s stressful to manage all of that individually.”

Enabled Delivery of User Requested Features 

Customers were demanding the ability to filter through their dashboards, but Fathom had been unable to support that. SingleStore solved this and allowed the team to ship a huge update to its software.

SingleStore has also laid the foundation to develop other new features that customers have been asking about for years. For example, Fathom recently shipped an incredibly flexible API for developers, which would not have been possible before. 

Faster Processing Time and Query Speed Optimizes Customer Experience 

Fathom’s query speed increased by more than 1,000X on average and is tens of thousands of degrees faster (than RDS) thanks to SingleStore. For some of its biggest customers and datasets, Fathom was able to go from 20+ minutes processing time to just 100ms with SingleStore, a staggering improvement of 12,000X. The data is now fully real-time and customers can filter on any field they wish. Fathom can also now use a single query to update and delete hundreds of millions of rows without using LIMIT and switched from UPDATE to INSERT to better support scaling, as appends perform better than updates.

“We took our analytics from slow, complex, bloated, rigid, and unscalable to flexible, fast, and powerful. Moving to SingleStore changed our lives,” said Ellis.

Optimized Infrastructure Spend for Sustainable Pricing 

“A huge part of running a software business is reaching this inevitable fork in the road: you can choose to raise prices when faced with increased costs or optimize your infrastructure spend for better margins and keep the prices the same. We chose the latter, as we felt it better serves our existing and new customers. We were able to do this by changing our technology stack to continue to be sustainable long term without continuously increasing prices across the board,” said Ellis. 

Fathom’s TCO was increasing, and DynamoDB’s cost was guaranteed to rise fast due to continued growth. By saving huge amounts of money with SingleStore, Ellis and team have been able to invest the savings in other areas of the business. Since Fathom operates as a zero-debt company, every dollar it frees up can be reinvested for compounded growth. 

Fathom has fixed monthly expense with SingleStore, and its cluster has the capacity to comfortably support huge amounts of traffic from enterprise and large SMB companies. It can now better serve its larger customers without needing to raise prices for its platform or suffering from unsustainably low margins.

Fathom is gaining rapid adoption and is on an accelerated growth curve today. Powered by SingleStore, Fathom can now scale limitlessly and deliver impactful customer experiences to power its privacy-centric website analytics platform, effectively disrupting the status quo in the space.

---

To learn more about how the visionary innovators at Fathom Analytics are gaining operational and competitive advantages with SingleStore, check out these blog posts by Jack Ellis:

• Building the World’s Fastest Website Analytics
• Making the World’s Fastest Analytics Even Faster
• Why We Ditched DynamoDB

SingleStore is helping companies compete and win across all verticals. Learn more →