Transform Pipeline - Clarity

Help
1

Exploring on transform to check on the technical feasibility of our usecase. Would like to get clarified on the below:

  • In the pipeline, we would like to put a logic like the data read from Kafka should be loaded in to SinglestoreDB only if a condition it satisfies and if not it should be loaded back to another Kafka topic. So the query is, can the Transform skip writing out records to stdout?

  • Also the below link 1, I could see that Transform Pipeline is not there in SinglestoreDB. But in the link 2, it is mentioned as it is not supported in SinglestoreDB Cloud. Can someone clarify on this please.

2

Hello Subash,

That is a typo there in the documentation. It should say that Transform is not available in SingleStoreDB Cloud in both pages. Thank you for catching that! I’ll update the documentation.

1 Like
3

Thanks @galley for the confirmation. Any thoughts on point 1 will be helpful.

4

Hi Subash!
For topic one, a transform reads from stdin to receive data from a pipeline’s extractor. After shaping the input data, the transform writes to stdout , which returns the results to the pipeline. The only possible destination for the data is through stdout back to the pipeline and into the SingleStore table. Data cannot be loaded into another Kafka topic through our CREATE PIPELINE ... WITH TRANSFORM mechanism - writing back to another Kafka topic would require application level logic.
Hope this helps. Please keep us updated on how it goes. :blush:

1 Like