S3 Data Ingestion Latency

Help Connectors and Pipelines
1

Hi,
I’m wondering what’s the typical data latency between a file written in S3 to the time it is ingested in Singlestore? (a file written every 15 sec with let’s say less than 10,000 records).

Would the latency be less if there records would be written to Kafka (every 15 sec) (MSK for example) and ingested via pipeline in Singlestore?

2

I’ll see about getting you some help with that question. Can you tell me the version of SingleStore you’re using and how it’s deployed?

3

I see in the documentation for S2MS, the CONFIG_JSON field batch_interval has a value of “2500”, which is milliseconds. I just tested this in S2MS with a pipeline reading from S3 and see that my default batch_interval value is 2500ms. PIPELINES · SingleStore Documentation