Kafka pipeline sometimes skipping messages

Help Connectors and Pipelines
1

Hello,

I have a table and a pipeline that inserts data from Kafka.

CREATE OR REPLACE PIPELINE player_audit
AS LOAD DATA KAFKA 'cluster/topic'
BATCH_INTERVAL 30000
SKIP DUPLICATE KEY ERRORS
INTO TABLE player_audit
FORMAT JSON
(
   @v_data <- data,
)
SET 
 ... fields set using JSON EXTRACT ...
;

Sometimes this pipeline misses events.

I stopped the pipeline, checked and saw 3 events in my kafka topic, but using test pipeline I got only 1 event in the pipeline.

After running ALTER PIPELINE player_audit SET OFFSETS LATEST; it suddenly worked OK.

It is important to note that this is development cluster - not prod - so it is not that powerful.

memsql 7.0.9
2 nodes that are not working that hard.

Any ideas why this might have happened?

2

can you show the output of

`SELECT * FROM INFORMATION_SCHEMA.PIPELINES_BATCHES WHERE PIPELINE_NAME='player_audit';`

and between which batches you executed the ALTER? was there any START/STOP PIPELINE DDL there as well?

similarly, you can verify during whether any particular kafka message offset was ingested in INFORMATION_SCHEMA.PIPELINES_OFFSETS.