What does offsets means.sometimes setting it to latest working and sometime earliest is working.
Is S3 pipeline can restore the data automatically?
1 Like
Hello @srilakshmiomkar, and thanks for using our forums!
The meaning of offset is a bit different between Kafka pipelines and S3 pipelines.
In Kafka, the offsets can set the starting point for reading messages for each topic partition. Setting them to earliest on a running pipeline means some data can get grab some data twice, whereas setting them to latest on a stopped pipeline means that current data in the topic could get skipped.
In S3, offsets are 0 and 1, and it indicates that the whole file has been read. Earliest and latest doesn’t quite apply in this context.
1 Like