Singlestore Kafka Pipelines fail with Timeout Errors

Help Connectors and Pipelines
1

Hello,

We have been facing timeout errors with Singlestore Kafka pipelines connected to Kafka endpoints on an Azure eventhub giving out error message as follows:

ERROR1934 ER_EXTRACTOR_EXTRACTOR EXTRACT…Cannot extract data for pipeline. Could not stream message (Are the Kafka brokers reachable from this node?)

The Pipeline batches run smoothly and suddenly we get a failed batch that runs for 30 sec. The failed batches start becoming more frequent till the Pipeline is force stopped based on the pipelines_stop_on_error=YES variable set.

We have been asked to apply recommended Kafka consumer configs to the Pipelines

and this is expected to solve the timeout errors that we are getting.

However, there are no ways that we have been able to figure out to apply these configs directly. Is there any way to apply them so that the Singlestore pipelines run with the recommended settings?

Thanks!

2

We have already played around with the engine variables listed under:

But, here we do not have all the configurations needed in place. And, the pipelines still fail.