Storing kafka keys in Singlestore pipelines

Help Connectors and Pipelines
1

Hi

I’ve been trying to create a singlestore pipeline to store kafka data into my table. It’s been running fine, and I have a dozen kafka fields being extracted into columns

But now I have an additional requirement to store the entire kafka message key into a text column. (no field extractions for the key, just have to dump the entire thing as a string)

I can’t seem to find a way to do this anywhere in the docs (this is the closest thing I could find, but it’s not what I want)

Any pointers on how to achieve this?

2

You could try inserting into a table with KAFKA KEY option.

The following example loads entire value to col_a and key to col_b
INTO TABLE table_name FORMAT JSON (col_a <-%) KAFKA KEY (col_b <-%);