Publish from memsql to kafka using pipeline

Help
1

Hi,
We’re using v7.6 and would like to copy data from 1 cluster to another .
In previous versions, we’ve used the export “insert … into” and import “load data local infile …”.
Would it be faster to copy using kafka pipelines?
Are there any examples how to publish data from memsql to kafka using a pipeline ?

2

hello Nabil, here is a great resource to publish SingleStore data into Kafka. https://medium.com/@VeryFatBoy/using-singlestore-as-a-kafka-producer-4d130d592c6d

Pipeline are commonly faster so you should be seeing performance benefits.

Regards,
Manish Kumar

1 Like
3

Hello, thank you for the above answer.

Regarding kafka pipelines, I was wondering if there was a less overhead way to publish as well as consume csv messages that have the same table format on both ends (instead of converting them into a json format)

Thanks,
Alexandre Cojot

4

Is this question about using Kafka or are you actually asking how to speed up the copy process of your database? If the latter is the case, you might want to look into REPLICATE DATABASE: REPLICATE DATABASE · SingleStore Documentation
It’s only available in Premium Edition though. Also, please read some comments about version compatibility: Replication Compatibility Between Different Cluster Versions · SingleStore Documentation

1 Like
5

Hi,
Replicate database is useful to sync a whole database. But, in our context, we only care to copy a subset of a large table from one cluster to another using the fastest option.