Can S3 pipeline read new file named same as deleted file which was already processed by this pipeline?

Help Connectors and Pipelines
1

Some external system exports data to S3 - but always to file with the same name.
Will S3 pipeline read this file after recreation or only ones after creation ?
Thanks!

2

Hello Ira,

Thank you for the question. With the current functionality of the pipeline you will have to recreate a new pipeline and use the same S3 file name for the newly created pipeline. We do not reuse existing pipelines once the file is deleted. Our engineering team will investigate making this work. There’s no timeline for delivering this yet.

You can create a new pipeline and pass on the same filename to mitigate the problem in the short term. This could be a reasonable alternative.

3

Hi @mkumar,
Thank you for info.

4

And when pipeline pulls files always from same buckets and some file in buckets will be recreated ?

5

I have a similar issue where file with same name is uploaded to S3 each business day. The old file is overwritten and new file is not recognized by running pipeline. If the pipeline verified against the unique S3 identifier instead of the filename, this could be resolved. Is there a place to make an official feature request to address this?

6

Welcome to the forum! :tada: Great first question!

While I get an expert to address your issue, can you tell me if you are on the self-hosted or managed service? Thanks

7

I am facing a similar issue while attempting to load a Parquet file from a GCS (Google Cloud Storage) bucket. Could someone please provide documentation or insights regarding this specific pipeline behavior? Additionally, I’m curious about how the SingleStore pipeline verifies whether a file has already been processed. Any information on these matters would be greatly appreciated.