Kafka pipeline - accessing message metadata (partitions, offset)

Help Connectors and Pipelines
1

Hey, when using kafka pipeline into stored procedure, is there a way to get each message partition or offset, and insert it into the target table?

thanks

2

Hi yarden! :wave: Thanks for your question. Would you be running on our managed service or self-hosted and your version number please?

3

Hey Maria, Thanks for responding,
self hosted,
version 7.5.12

can upgrade if needed for this feature

4

Hi Maria, same question self hosted, version 7.5.12

5

Hi yarden & will.smith!!

All metadata about pipelines, including partition & offset information, can be found in the following information schema tables.

For this particular use case:
PIPELINES_BATCHES & PIPELINES_CURSORS have good information.

I hope this helps. Would love to hear how this works out for ya’ll! :smile_cat: