Kafka pipeline into stored procedure - sequence

Help Connectors and Pipelines
,
1

Hi,

I have a pipeline into stored proc.
All our tables are columnstore tables.
The pipelines are set with MAX_PARTITIONS_PER_BATCH to 1.
We need to strictly process messages in the sequence. To support idempotent we check each message before inserting or updating. Ignore if it is already there.

Here is the message sequence in the topic.
Message 1 :- insert
Message 2 :- Update
Message 3 :- Update
Message 4 :- Delete

The above 4 messages can be posted within 5 milli secs.

Singlestore does not invoke the stored proc with the same sequence and we get message in the sequence below or some other sequence other than the kafka message order.

Message 1 :- insert
Message 2 :- Update
Message 4 :- Delete
Message 3 :- Update

DISABLE OUT_OF_ORDER OPTIMIZATION is not supported for PIPELINE into stored proc. Is there any other setting to maintain the order, Or, Is there any workaround?

2

Consider just inserting the data as rows in a queue table of some kind with a timestamp or sequence number to designate order, then have application code process them in order. That could be external application code or a stored procedure.

There is no native, built-in workaround.