How To Stream Real-Time Data Into SingleStore From Estuary Flow with Dekaf

Starting today, developers can integrate hundreds of real-time data sources into their SingleStore database using Estuary Flow’s new Kafka-API compatibility feature, Dekaf.

Dekaf acts as a bridge that allows SingleStore to seamlessly consume real-time data streams from Flow as if it were communicating with a Kafka broker.

An overview of Dekaf

Dekaf is Estuary Flow’s Kafka API compatibility layer, designed to make integration with Kafka-based systems seamless. By mimicking Kafka’s API, Dekaf allows any system expecting Kafka as a source or sink to interact directly with Estuary Flow — while Flow itself handles the complexities of data capture, transformation and streaming from a wide range of data sources.

With Dekaf, developers can use Estuary Flow as a drop-in replacement for Kafka without modifying existing Kafka consumers. This means even though Flow may be connected to databases like PostgreSQL or MongoDB®, it presents data in a Kafka-compatible format, making the integration process much simpler and faster for systems like SingleStore.

Key features of Estuary Flow and Dekaf

Fully integrated pipelines. Flow simplifies data integration by enabling you to create, test and change pipelines that gather, modify and consolidate data from multiple sources.
Change Data Capture (CDC). Always-on CDC that replicates in real time with exactly once semantics, backed by cloud storage in your own private account.
No-code connectors. With 150+ pre-built connectors for popular data sources and sinks — like databases and message queues — Flow reduces the need for custom connectors. This speeds up data pipeline deployment and ensures tooling consistency across systems.
100% Kafka consumer API compatibility. Dekaf fully implements Kafka's consumer API, making it indistinguishable from a traditional Kafka broker.

How does Dekaf work with SingleStore?

When using SingleStore’s Kafka pipeline, which allows for fast, exactly once processing to ingest data, Dekaf serves as the intermediary between SingleStore and Estuary Flow. Here’s how it works:

Pipeline setup. SingleStore’s Kafka Pipeline is configured to communicate with Dekaf just as it would with a Kafka broker. Estuary Flow acts as the data producer, streaming data into topics that Dekaf exposes to SingleStore.
Topic management. In Estuary Flow, each capture is organized into logical topics, similar to Kafka partitions. SingleStore subscribes to these topics through Dekaf, ensuring data is consumed in real time.
Schema handling. Dekaf also handles schema management through integration with schema registries. It ensures SingleStore receives data with the correct structure, minimizing the need for manual schema mapping.
Data delivery. As changes happen in source databases (like MongoDB or PostgreSQL), Estuary Flow captures these changes and streams them via Dekaf to SingleStore. This provides a near real-time view of the data in SingleStore — perfect for applications requiring up-to-date information for analytics or operational purposes.

By using Dekaf, SingleStore users can integrate real-time data pipelines from a variety of sources, leveraging the power of Estuary Flow while maintaining the simplicity of Kafka-like operations.

Get started with SingleStore and Estuary Flow’s Dekaf

Create PostgreSQL capture in Estuary Flow

Now that PostgreSQL is up and running with CDC enabled, the next step is to create a capture in Estuary Flow.

1. Create a new capture. Go to the Estuary Flow dashboard and create a new PostgreSQL capture.

2. Configure the endpoint

3. Define the collection

In the collection configuration screen, select the sales table for CDC. This will create a collection in Flow that continuously captures changes from your PostgreSQL database.

4. Start the capture. Press “Save & Publish” to initialize the connector. This kicks off a backfill first, then automatically switches into an incremental continuous CDC mode.

5. Set up SingleStore Kafka Pipeline

CREATE TABLE test_table (customer_id NUMERIC, product_id NUMERIC, quantity NUMERIC, 
sale_id NUMERIC, sale_date VARCHAR(255), total_price NUMERIC, unit_price NUMERIC);

CREATE PIPELINE test AS
        LOAD DATA KAFKA "dekaf.estuary.dev:9092/dani-demo/postgresdemo/sales"
        CONFIG '{
            "security.protocol":"SASL_SSL",
            "sasl.mechanism":"PLAIN",
            "sasl.username":"{}",
            "broker.address.family": "v4",
            "schema.registry.username": "{}"
        }'
        CREDENTIALS '{
            "sasl.password": "Your Estuary Refresh Token",
            "schema.registry.password": "Your Estuary Refresh Token"
        }'
        INTO table test_table
        FORMAT AVRO
        SCHEMA REGISTRY 'https://dekaf.estuary.dev'
        ( customer_id <- customer_id, product_id <- product_id, quantity <- quantity, 
sale_id <- sale_id, sale_date <- sale_date, total_price <- total_price, unit_price <- 
unit_price);

6. The last step is to test out your pipeline!

TEST PIPELINE test LIMIT 1;

START PIPELINE test;

select * from test_table

The integration of SingleStore and Estuary Flow, enabled by Dekaf, marks a significant leap forward in real-time analytics capabilities. By combining SingleStore’s processing power with Estuary Flow's extensive source support and Dekaf's integration, organizations can implement robust, scalable and efficient real-time analytics solutions with unprecedented ease.

Try SingleStore free today.