Streaming data out from DB to Kafka

Help Connectors and Pipelines
1

Hi, I saw we can export data into Kafka and many other targets using select into query

This query we have to run periodically to get the data in to Kafka. I’m wondering if this can be run in the background like a normal pipeline.
For example all the new records matching the query are exported to Kafka by this and periodically running is not required

Thanks for your answers

2

I think you’re asking for a continuous export of new data matching a SingleStore query to Kafka. That’s not supported.

I’d recommend creating a cron job or customer application to run SELECT … INTO KAFKA periodically, and grab new data (by filtering on a datetime column, say).

3

thanks for the reply hanson.
Any plans in the future to add export as a stream feature? It’s not a blocker at all, but would like to know if there will be plans to add it as a feature.

Thanks,
Venura

4

We’re looking at doing change data capture (CDC) out of SingleStore at some point in the future, but I can’t give you any date estimate on that. Similarly for triggers. Both of those can be considered a way to get an export stream.

5

Hi, is this feature supported now? I have a use case where I wish to stream general query log (only SQL queries) from Singlestore to Kafka.

6

Hi @rahilsanghavi. CDC Out is available in Preview: