Multiple pipelines in the same S3 bucket

Help
,
1

Hi Team,

We are currently deploying Singlestore and Minio. Currently we are setting up each bucket for each category and within the category there are few paths. We are loading files in each path using pipeline to Singlestore.

CREATE PIPELINE pipeline_cat_a_1
AS LOAD DATA S3 ‘cat/a/1’ … INTO TABLE cat_a1;

CREATE PIPELINE pipeline_cat_a_2
AS LOAD DATA S3 ‘cat/a/2’ … INTO TABLE cat_a2;

However, in the Studio, the active queries for the pipelines are running one by one not in parallel. We would like each pipeline ran concurrently. What is the best way to achieve this? Is one pipeline for one bucket only?

Appreciate for your feedback and suggestions.

2

Hi,

Pipelines are concurrent and not blocking each other, you might be seeing queries from pipelines sequentially in active queries by coincidence or the pipelines batches are scheduled differently.

You can see the status of batches that are progressing in
information_schema.batches_summary table