MemSQL Pipeline error extracting data from Kafka

Help Connectors and Pipelines
1

I am hitting an error when running MemSQL pipeline to load data from Kafka -

Cannot extract for pipeline — because ‘FAILURE’. Truncated stderr:
Invalide message offset, consumed outside of range.

Kafka topics has 5 partitions and MemSQL has 8 partitions. Could that be an issue?
SET OFFSETS EARLIEST was done before starting the pipeline. How can I debug this?

Thanks in advance.

2

Hello wnyd2d. Thank you for trying out Kafka pipelines!

Unequal number of partitions between MemSQL and Kafka usually does not cause an issue.

The error you are seeing can happen if MemSQL tries to ingest a range of messages, but Kafka has already deleted them. FYI, this behavior is changed to allow the pipeline to continue running in MemSQL 6.8.9. SingleStoreDB Cloud · SingleStore Documentation

It might also be useful to review the maximum lifetime of a message in the Kafka topic.

If upgrading the server is not possible, could you please provide your server version. It is possible you could restart the pipeline, without losing the offsets.

3

Hi Mkobyakov, thanks for looking into this.
The MemSQL version is 6.7.15. Will try upgrading the server to newer version.