Arcules Scales Video Surveillance and Analytics Platform with SingleStore to Support Thousands of Global Users

Consumer-grade security systems made the leap from analog to cloud-enabled relatively quickly, but enterprise-grade technology lagged behind. These organizations found themselves wasting time on applying software updates, maintaining remote access, worrying about risk, and ripping out cameras when it switched systems. 

Arcules, owned by the Canon Group, has as its primary goal to make the world a safer place. Arcules got its start in an incubator program in 2013. This diverse and passionate team of security and cloud experts shared a desire to create the world’s most user-friendly, affordable, and flexible security surveillance system and insights solution.

The platform Arcules developed offers an intuitive, cloud-native solution that unifies and makes sense of the data from video  surveillance systems for security and beyond. The Arcules platform is device-agnostic, with a plug-and-play experience that uses a small edge hardware appliance to autodiscover and connect supported devices like IoT cameras, sensors, and access control panels to the platform. Users can view all their remote sites in one interface through the web interface or native Android and iOS mobile apps, making it ideal for multi-site distributed services including franchises such as WeWork. 

Arcules views its cloud platform as not only the future of security video, but also an all-inclusive analytics tool offering meaningful operational insights using the data from cameras, sensors, door controllers, and other devices. 

Challenges/Goals 

Many of the typical video surveillance management systems today for enterprises lack the concept of viewing all of its sites in a single, cloud-based package. Arcules caters to multi-site distributed services like franchises, where organizations have many locations without a single pane of control, by providing video surveillance-as-a-service. However, that’s not the only use it has in mind for its platform.

While the main focus of its solution is security and surveillance, Arcules wanted to expand the system to offer analytics from video data to get more value from the data. To do so, it needed all-new architecture. 

Benjamin Rowe, Cloud and Security Architect at Arcules, explained, “We were growing as a company, so we had to do something about scalability of the platform. A lot of what we had built originally as a startup was just to get us off the ground and get our proof-of-concept in production. Two to three years ago, we started to see adoption of the platform, so we had to start web-scaling everything. The entire system needed to infinitely scale.”

Benjamin Rowe, Cloud and Security Architect, Arcules

Without this scaling capability, the user experience would suffer. The structure of the platform was built with microservices, but the underlying database was struggling. 

“We were already microservices-based, so we were on a good footing, but a lot of our database technologies behind the scenes weren’t ready for the scalability that our microservices provide,” added Rowe. 

As part of a seamless user experience, Arcules needed a video index to allow the system to intelligently decide where the video data should come from. If the data is in the cloud and the user isn’t at the local site, then it can be accessed from the cloud. If the user is on-site, then they can access the local data. Arcules wanted to make the experience a lot cleaner and faster for the user, with the video data source abstracted away so it still feels like an on-premises security system.

“We want our customers to experience very little lag,” said Rowe.

Technology Requirements 

To enable real-time analytic processing of the data and a seamless user experience, Arcules needed a modern database that could deliver speed and scale at a cost-effective price.  

Rowe articulated Arcules’ main concern. “The biggest problem we had was scalability. We’d hit the vertical scaling limits of the databases before.” 

Arcules’ other concerns included: 

• Providing the correct solutions for customers, supporting its needs for fast data and real-time insights.
• Offering the right fit for customers to meet the surveillance requirements for enterprises. 
• Creating a seamless experience for the user from a product perspective, to deliver as-good-as on-premises latency for its surveillance systems.  

The ideal database technology needed to deliver: 

• Real-time analytics and gap and island analysis on all video datasets 
• Ad-hoc requests of certain timeframes to generate data for the user
• Support for Google Cloud (GC) 
• Processing speeds capable of returning results on 30 days of video data within 10 seconds for near-real-time accuracy 
• High availability and multi-region support

Why SingleStore 

When Arcules originally launched its platform, it used MySQL as part of its bootstrapping process. Arcules switched to PostgreSQL for better security and data segregation, using Google Cloud SQL for a fully-managed relational database. It  developed its platform using containerized microservices on Google Kubernetes Engine (GKE), connecting to the data through Cloud SQL Proxy sidecars. However, this setup could not keep up with Arcules’ success, as PostgreSQL had a lack of horizontal scaling.  

The team started exploring other database options four years ago. PlanetScale didn’t work for its use case, and while ElasticSearch solved the use case issue, it was cost-prohibitive. Then SingleStore appeared on its radar.

“We had worked with other scalable databases, but we always ran into an issue that they were very good for either writing or reading, but not both. The other problem we ran into was cost. The biggest reason we picked SingleStore came down to cost. The other databases couldn’t perform the way we wanted them to cost-efficiently,” said Rowe.  

SingleStore is built for data intensity and provides a robust feature set capable of powering and scaling Arcules’ platform:

• Run anywhere capabilities support Arcules’ deployment on its on-premises Kubernetes cluster and GC. 
• Universal storage offers a patented single table type for transactions and analytics, reducing latency by bringing these workloads together. 
• Compatibility with ANSI SQL for a smooth transition from PostgreSQL. 
• Fast ingestion for keeping up with continual surveillance video feeds. 
• Pure speed for fast transactions, queries, and analytical processing. 

Rowe highlighted one of his favorite features: “One of the things we like about SingleStore is that it heavily compresses the data. The raw data ingested into the database versus the actual data that SingleStore stores is radically different because we compress it by at least 90% all the time.”

Solution 

Arcules uses SingleStore to power its video indexing capabilities to support real-time video indexing, and fast data to improve the user experience and expand the platform’s capabilities. It opted for the SingleStore Self-Managed solution, hosting it on GC. 

Arcules runs off of three SingleStore production database clusters using four nodes, located in the United States, Europe, and Japan. They are high availability, with active failovers. These databases are supported by several development clusters. 

The data moves from the user’s cameras and IoT devices through the Arcules edge appliance to the Arcules gateway, which is its on-premises Kubernetes cluster. The data gets converted into a specialized format, and the metadata is propagated via Google Bucket in GC. This process auto-triggers to feed the data via a pub/sub model API into SingleStore. The index allows Arcules to intelligently decide where to grab the data from: either the cloud or local systems.

“We feed it from both the gateway where the data originates and the cloud. If there are any delays in the cloud, the platform automatically knows to go to the gateway to get the latest data, so it’s effectively real-time,” detailed Rowe. 

Arcules has a data-intensive platform, with 30 terabytes (TB) of data attached to the disks today, with 15-20 TB being utilized on a 30-day basis. A typical user channel has two-and-a-half gigabytes (GB) of video metadata. 

Users will request the data for specific timeframes. Arcules does gap and island analysis within SingleStore’s query language to generate data for the user, allowing them to quickly navigate through the video content without having to scrub the data. 

As a multi-tenant, global solution serving users in 15 countries, it took a few months for Arcules to determine the right shard key to ensure that the data was distributed properly.

“When running certain types of queries, such as gap and island analysis, you’re trying to keep the data on the same node, so they don’t have to jump the nodes and go to the aggregators. That’s where the more intelligent time bucketing concept comes into play. We’re using it quite heavily inside SingleStore to do the gap and island analysis,” explained Rowe.  

The queries that Arcules runs to produce this time bucket analysis are complex and resource-intensive. It’s a full analysis that looks at the patterns within certain time boundaries. It modifies the query to smooth out the video to account for any packet drops. If the time between one video and the next is within a few hundred milliseconds, it gets ignored. Otherwise, the system performs other processes on it, such as generating heat maps, verifying door access, and counting people in a room.

From an ingestion perspective, Arcules averages more than 100,000 inserts per second, but it varies. For example, the ingest and user concurrency are lower overnight because there’s not as much video being recorded. It can also see peaks and troughs in the usage because of SingleStore.

Outcomes

Arcules has gained serious advantages in real-time performance, scalability, and user experience as a result of choosing SingleStore:

Real-Time Performance Gets Video Data to Users On Demand

“Getting that video data fast is ultimately what our users want, and it’s the lifeblood of Arcules. We’ve achieved that with SingleStore.” explained Rowe.   

Users have a seamless experience as though the data is local because of SingleStore’s speed. Ingest speeds range from thousands to millions of rows per second, while the system averages approximately 250,000 inserts per second. 

Both data query results on an hour’s worth of data and resilient ingestion had sub-second response times. For 30 days of data, with 2.7 million records, results arrived within 10 seconds. 

Robust Scalability Supports Thousands of Concurrent Users

“SingleStore has saved us time from a scalability perspective, as it’s easy and fast to deploy when we’ve spun up new clusters.  It was the right choice. The cost in relation to the scaling with SingleStore seemed like the best bang for the buck,“ detailed Rowe. 

For the video feeds, it is ingesting terabytes per day and has 30TB attached to the disks. However, the data it stores per channel is much lower at 2.5GB, thanks to the more than 90% compression rate of SingleStore. 

As Arcules grows, it’s now able to support thousands of users accessing the application simultaneously.

Off-the-Charts Performance Improvements (20,000%) 

Arcules was one of the first startups to enter the surveillance market with a cloud-native, real-time solution, and the 20,000% performance improvements that SingleStore brought to the system had people shaking their heads in disbelief. 

“Our users didn’t believe it was that fast, and with SingleStore, we’ve encountered that multiple times. Given that we were one of the first to enter the market with this kind of concept, users thought there was no way that was happening in real time. One of our field CTOs said we had to adjust the performance improvement numbers down from 20,000% to 1,000% because our users didn’t believe the actual number,” Rowe explained.

Arcules is looking towards the future for its data infrastructure with SingleStore. It plans to add more analytical data and other metadata types that may tie back to video in these database clusters, using them like a time series database.

“As we move forward, we’re starting to put our analytical data in SingleStore. For example, we’ve started  adding detection, people, vehicles, colorization, and other metadata types that tie back to video,”  Rowe said.

Arcules expects to feed 100% of this analytical data in SingleStore within a year. 

Another exciting plan on the horizon for Arcules is incorporating AI for basic array searching. For example, a user could say, “I need a red car and it needs to be within X area of my footage,” and they would receive matching results.

---

SingleStore is helping companies compete and win across every vertical. Learn more →