“In our business, you innovate, or you die. We had to find a company that innovates. We found that with SingleStore." Sandip 'Sandy' Mehra, CEO, Telgoo5 Technology
Imagine a business that provides customer service-as-a-service to a wide range of clients worldwide. It develops a robust, in-house digital infrastructure to serve a wide range of customer needs. Eventually, the digital infrastructure becomes so capable it decides to offer this as a product in its own right; the new offering is a huge success, and it runs on SingleStore.
No, not AWS (a SingleStore partner, and one of the public cloud platforms for SingleStore Cloud). We’re talking about Telgoo5, a long-standing and highly innovative SingleStore customer.
It all began at Vcare, which started out providing call center support as a service to a wide range of customers. Now Vcare has spawned a new business: a real-time charging platform for mobile telecom operators. The system, Telgoo5, provides mobile virtual network operators (MVNOs), who sell mobile access to their customers, with ultra-fast connections to the largest communications service providers (CSPs) including AT&T, T-Mobile, Verizon, and more. It also keeps Telgoo5 ahead of its competitors such as CSG, NEC/NetCracker, and Oracle. Global research firm MarketsandMarkets forecasts the MVNO market to grow to nearly $90B by 2024.
What makes these processes data-intensive is that an MVNO must verify a multitude of things FAST to ensure its customers' calls connect when they place a call. It must verify that a customer has sufficient balance available to make a phone call, send a text message, or use apps, and it must also verify background data requests from various apps including messaging, mapping, GIS, and navigation. If verification does not occur instantaneously, in 250 milliseconds or less, the data connection fails, calls don't connect — and MVNO customers churn.
“In the current pandemic crisis, the two sides of our business are behaving differently,” said Sandip 'Sandy' Mehra, Chief Executive Officer, Telgoo5. “On the Vcare side, where we provide customer support services, we are waiting and watching, to see how customer demand changes during the crisis. But on the prepaid mobile side, we anticipate steady demand, or even growth. Communication is more and more important, and prepaid mobile is a flexible alternative that customers like — especially with the performance that SingleStore is helping us to provide.”
Telgoo5 used to rely on a MySQL-based solution running in five data centers around the world. CSPs sharded the database manually; partitioning problems and the need to reorganize data frequently caused downtime. The system was often missing or barely meeting the 250ms window, causing connections to drop. This created unhappy MVNOs and MVNO customers, and reflected poorly on Telgoo5; and while this problem was shared by competitors, Telgoo5 was missing a major opportunity to stand out from the crowd.
Solution: SingleStore
Four years ago, SingleStore pitched Mehra and team on a SingleStore-based solution and today they are running Singlestore Helios on AWS. Telgoo5 has discontinued using those five physical data centers, each of which was a potential source of downtime and other problems. Missed connections and downtime have gone from “all too common” to “almost unheard of” as data connections often occur in as little as 35ms, more than 80% faster than required.
Telgoo5 uses its new infrastructure to run machine learning models and AI programs monitoring customer success and suggesting interventions, such as special offers and discounts on additional service levels, to help keep customer loyalty high. (Yes, as in our joint blog post with Fiddler.ai, Telgoo5 is using AI to reduce churn.) Also, as 5G deployments grow, the Telgoo5 system smoothly uplevels to the much higher data volumes and much shorter response times needed by 5G, with nary a hiccup.
Telgoo5’s telecom customers have more than tripled their business volume with no impact on uptime or performance. Telgoo5 simply scales its infrastructure smoothly as business volume grows, and confidently reaches out to new customers, growing its solutions both horizontally and vertically to provide virtually unlimited charging capability.
Telgoo5 is facing different challenges in its two lines of business. In the call center and related support services, (v)WeCare Technology's original business, disruption caused by the coronavirus upended business planning worldwide, and the need for call center support was reduced. On the online charging side, however, businesses are seeing an ever-greater need for mobile communications. Prepaid mobile service, which is what Telgoo5 supports, provides a flexible and cost-effective alternative to subscription plans. With its backbone running on SingleStore and AWS, Telgoo5 is well equipped to maintain and grow its share of this competitive market.
The Online Charging Systems Challenge
Online charging systems are used to provide data services for prepaid mobile users – the most common form of mobile phone billing in the developing world, and increasingly popular in economically developed countries. When a phone call is made, or a data connection is requested, metadata about the request – not actual voice or data content – goes out to the online charging system.
The online charging system has a brief window, typically 250ms, to either make the voice or data connection, or fail. A mobile phone may instantly make 60 simultaneous requests for data. Requests come from mapping apps such as Google Maps, messaging programs such as WhatsApp and Skype, email, built-in text messaging, and many others.
The data requests are made in 5MB chunks. For each request, if the connection is not made in time, or if a voice call or data transfer fails in progress, the phone service has failed the customer.
It doesn’t take many failures for customers to demand refunds, cancel service, switch suppliers, etc. At worst, a customer can develop the belief that prepaid services in general simply don’t work well enough, taking them entirely out of the market that Telgoo5 serves.
Each transaction is supported by a customer transaction report (CTR), a record of the interaction. Hundreds of millions of CTRs are created and logged in a typical day. These CTRs support business on the scale of tens of millions of dollars a month.
As phone call and data requests are fulfilled, the customer’s prepaid account is decremented for the voice call or data transmission. As certain data limits are reached, data speeds may be throttled until more headroom is purchased for the account.
“We expect a lot from a database,” says Mehra. “Not only are we making requests and responding; we are creating plans, adding subscribers, removing subscribers, moving subscribers from one plan to another. We are executing real-time transactions, but beneath the surface, the database is doing so much more. You can think of it as creating virtual plans, as the customer goes about their day.”
SingleStore is a uniquely capable, fully distributed relational database, with full SQL support and high concurrency. (That is, the ability to scale to support many thousands of queries per second, from any number of requestors.) Before using SingleStore, Telgoo5 was stuck with the same bad choices as other providers:
- Manual sharding. Many customers of MySQL and other single-process relational databases shard their database manually, then try to maintain and balance the shards through their operations staff.
- Paying a provider for manual sharding. Other customers pay a large database company for complex, integrated, hardware-plus-software solutions. They are basically paying an external company to shard and manage their single-node database for them.
- No cloud. Many providers offer no cloud solution, or limited solutions. This leaves customers building or leasing data center capabilities, with both internal and external providers finding it difficult to maintain acceptable levels of uptime and responsiveness.
- Missed SLAs. All of the existing database solutions strained to meet the required 250ms SLA with any regularity or reliability, causing problems for telcos and their end customers, who constantly suffered from bad connections.
As an example of the kinds of problems that other operators suffer frequently, customers in the Philippines suffered a nearly eight-hour outage last year. Roughly 60 million subscribers suffered complete loss of service.
Before using SingleStore, Telgoo5 maintained five data centers just to run their manually-sharded MySQL database in these on-premises locations.
| Before SingleStore/ Issues | With SingleStore/ Results | |
|---|---|---|
| Technical solution | - Manually-sharded MySQL - Constant architectural and operation difficulties | - Fully scalable SingleStore - No architectural issues and ease of operations |
| Infrastructure | - Five leased data centers - Failure to meet SLAs; operational and management hassles | - Distributed on AWS - Easily meeting SLAs; much easier operations and management |
| Performance | - Frequently failed to meet 250ms connection window - Dropped calls, dropped data connections, slow responses | - Regularly achieves 35ms connections - Completed calls, completed data connections, fast responses |
| Development & testing | - Development and test environments very different from deployment - Constant hassles in services rollouts and trouble-shooting | - Development, test, and deployment environments very similar / identical - Much easier services rollouts and trouble-shooting |
| Business scale | - Difficult to maintain stable presence in challenging business | - 3x growth in business scale and many likely opportunities for growth |
| Flexibility | - Very difficult to make changes | - Easy to make changes for arbitrarily challenging environments |
Table 1. Technical, operational, and business prospects all improved with SingleStore.
How Telgoo5 Stands Out
SingleStore, by contrast, has had truly cloud-native software for many years – since before the term “cloud-native” even caught on. SingleStore offered Mehra and team a quite different solution:
- Telgoo5 moved to SingleStore, a fully distributed relational database, with no need for manual sharding. Operational costs dropped sharply.
- Telgoo5 runs SingleStore on Linux, on commodity hardware, with low server costs and easy maintenance, and easy scalability. This gives Telgoo5 nearly endless choices for development, testing, deployment, and backup.
- Because SingleStore supports ANSI SQL and is MySQL wire protocol-compliant, Telgoo5’s applications could more easily move from MySQL to SingleStore. These widely-used standards allowed Telgoo5 to choose from a wide range of business intelligence tools.
- Telgoo5 now runs entirely in the cloud, running on SingleStore in selected AWS regions. This solution delivers far greater reliability, flexibility, and uptime, with much less operations oversight.
“We use all rowstore tables – in this business, with the demanding response times, everything is in-memory,” said Mehra. “Our original customer is so happy – knock on wood,” says Mehra. “They’ve had almost no downtime in the past three years, while their business has grown exponentially. Downtime is gone, and response times are great. We are getting new leads from potential customers worldwide – all to be hosted on SingleStore and running in AWS.”
All of this is improving Telgoo5’s market share and prospects in the telecom billing and revenue management market – a market projected to nearly double in size in the next five years, reaching nearly $20B in size.
“With SingleStore, we’re able to deliver, and outperform our customer’s expectations. We have no single point of failure. We complete transactions well within the SLA and give amazing service. We consistently deliver outstanding services to our clients.”
How the Architecture Works
The Telgoo5 platform brings in a range of inputs into a series of data stores, which then power a wide range of business intelligence tools, reporting tools, and apps.
There are three inputs, two of which go to SingleStore. The Diameter Routing Agent and HTTP requests both go directly into SingleStore, with HTTP requests processed through a high availability (HA) architecture, and also sent to MySQL. Performance Co-Pilot (PCP), an open source framework for monitoring system performance, sends input directly to MySQL.
Data then flows across four data stores:
- SingleStore handles transactional data processing and data analytics, sending results to three BI tools – Qlikview, RapidMiner, and Tableau – and on to MySQL.
- MySQL handles reporting and processing for the PCP dump, while data from both SingleStore and MySQL is used for reporting, data analytics, and predictive modeling using SQL, Python, and R.
- Processed data is then sent to AWS Aurora for distribution to reporting and email output, using Python and R.
- AWS Amazon S3 is used for long-term storage.
This amalgamated data processing system supports a wide range of critical reports, updates, and predictive analytics models. SQL, Python, R, and Ruby are all used for data processing, analytics, and updates. QlikView, Tableau, and RapidMiner provide BI capabilities. And an internally created dashboard called R Shiny is integrated to the SingleStore database, as are email outputs and XML data processing, coded in Python and R.
Business Results from the New Architecture
“As a result of this new architecture, the size of our original business relationships have changed. Our telecom company customers are able to grow their customer bases, and end customer satisfaction has increased, because their services hardly ever go down. They not only increase revenues – they become more profitable as well.”
“We are able to track multiple indexes for each of our customers. If any measurement falls short, we reach out and let them know. They are very happy with our business intelligence (BI) capabilities and our proactive reporting,” says Mehra.
When Telgoo5 first engaged with SingleStore, they felt they were buying into the future. And SingleStore has continued to help deliver futuristic capabilities to Telgoo5 ever since.
Today, Telgoo5 uses machine learning and AI, running against the SingleStore database, to monitor quality of service – QoS, in telecom parlance – and to detect potential customer churn. Specific, targeted, cost-effective interventions are made available, helping to keep QoS high and churn low.
“The big thing in telecom is, how do you predict which customer is going to churn? For our competitors to process, say, 100 million CTRs – less than one day’s worth – might take them several hours. We can do this in minutes. Both we and our customers can do deep dives on the data.”
| Before / MySQL | After / SingleStore | |
|---|---|---|
| Database | MySQL | SingleStore |
| How to accommodate volume | Hand-sharding and manual partition management, with downtime | Easy scale-out without downtime |
| Data infrastructure | Five contract data centers, with downtimes, operational issues, etc. | AWS availability zones worldwide |
| Cloud-native? | Not available when decision was made | Yes (now has Kubernetes Operator) |
| Meeting 250ms SLA? | Often missed, resulting in failed data connections | Nearly always achieved; many responses in ~35ms |
| Missed connections and downtime | Common; competitor in Philippines recently had nearly 8 hour downtime | Vanishingly rare |
| New customer/capacity setup | Months of advance work | Smooth scaling with demand as needed |
| Cost basis | Big capex investments before capacity can come online | Smooth increase in opex as business volume increases |
| Time required for routine reports | Many hours | Several minutes |
| 5G readiness (“exponential increase” in data volumes & velocity) | Extremely challenging | Easy |
| Machine learning/AI responsiveness | Not real-time; offline/batch only | Near-real-time / meets SLAs |
| Competitiveness | Average | Exceptional |
Table 2. Before (MySQL) and after (SingleStore) comparison for Telgoo5.
How Telgoo5 is Growing Their Business
With all this information, the telecom provider cost-effectively grows their business; and Telgoo5, as the online charging system provider, using SingleStore as their core data platform, grows right along with them. Telgoo5 is also using their initial success and positive client feedback to make aggressive proposals for new business worldwide.
As Sandip Mehra describes it, “Previously, with our data center-based solution, we had a very high Capex (capital expenditure) for any new client – and we had to make the investment up-front, before we could even get started. It took three to six months to grow a client’s infrastructure; now, it’s instant.”
“With SingleStore,” continued Mehra, “running in the cloud, we have a much more manageable Opex (operating expenditure), which starts small and scales smoothly with demand. The money we save there goes into client satisfaction; we offer excellent support, and easily scale the backend horizontally and vertically, as needed. With our original client, we’ve smoothly supported them as they’ve tripled their business, with great customer satisfaction. And we do all this while maintaining excellent response times.”
“This puts us in a powerful position, compared to our competition. They are using traditional databases and have high fixed costs.”
5G represents a big effort across the telecoms world. It generates far more data – an order of magnitude greater, or more – for a given operation, and demands much faster response times. “5G exponentially increases the amount, and the speed, of data that are flowing in,” says Mehra. “The need for real-time charging and customer rating increases, and we have to do it a lot faster.”
“In our business, you innovate, or you die. We had to find a company that innovates. We found that with SingleStore.”
Conclusion
Like everyone else, Telgoo5 is facing rapid change in these challenging times. However, their move to SingleStore, to power their prepaid mobile phone charging solution, put them in much stronger shape for any circumstances. Telgoo5 is now prepared to maintain and grow this new line of business.
If you are interested in knowing more about SingleStore for your own business, you can try SingleStore for free or contact us.
