Kafka Pipeline consuming a transactional kafka topic

Help Connectors and Pipelines
1

Hi,

I’m trying to setup a very simple kafka pipeline that consumes a topic which is populated with a transactional producer (ie the topic contains transaction markers too).

CREATE OR REPLACE PIPELINE mypipeline AS LOAD DATA KAFKA ‘broker/topic’ INTO TABLE table`;

When testing the pipeline, I’m getting the following error:

ERROR 1934 ER_EXTRACTOR_EXTRACTOR_EXTRACT: Leaf Error (127.0.0.1:3306): Leaf Error (127.0.0.1:3307): Cannot extract data for pipeline mypipeline. Stderr:
2019-10-04 10:15:02.652 Waiting for next batch, 255 batches left until expiration.
2019-10-04 10:15:02.652 Batch starting with new consumer.
2019-10-04 10:15:02.820 Partition EOF stremaing offset from Kafka, resetting to earliest known offset
2019-10-04 10:15:02.922 invalid message offset, consumed out of order

If I try the very same pipeline with a topic that’s identical to the previous one but no transactions, it works fine.

Does the kafka pipeline work with transactional topics at all? This issue may be similar: apache kafka - Pipeline is not ingesting data into memsql table using procedure - Stack Overflow

Thanks,
Laszlo

2

Hello Laszlo. thanks for trying out MemSQL pipelines.

at this time, MemSQL uses non-transactional Kafka consumers, and keeps its own records of partitions and consumed offsets for each topic. Using a transactional producer is likely incompatible with this.

Could you describe a bit more about your use-case? How do you use Kafka transactions?

3

Hi,

We have complex streaming pipelines using kafka EOS (transactions) and we simply want to ingest some of the topics in memsql. All we need is a consumer’s capable of handling the “isolation.level”: “read_committed”, nothing else really. Not sure if this is the problem I have, but potentially may be, because I tried to ingest 2, almost identical topics: one had transaction markers in it, other didn’t, and the former failed.

Thanks,
Laszlo

4

Same problem here. 2 topics, same info on both. One of them fed with a transactional producer, the other not with a “normal” producer. The transactional topic fails with the “message offset” error.
Are there plans to support this?