Pipeline from S3 parquet files to stored procedure

Help
1

Hi Guys,
Is it possible ? Can somebody give working example ?
Thanks!

2

Hello,

Here is an example of such a pipeline that I took from our real-time digital marketing application:

CREATE PIPELINE `locations`
AS LOAD DATA S3 'singlestore-realtime-digital-marketing/v2/100k-2p/locations.*'
CONFIG '{ \"region\": \"us-east-1\" }'
CREDENTIALS <CREDENTIALS REDACTED>
BATCH_INTERVAL 2500
MAX_PARTITIONS_PER_BATCH 2
DISABLE OFFSETS METADATA GC
INTO PROCEDURE `process_locations`
FORMAT Parquet
(
`subscriber_id` <- `subscriberid`,
`offset_x` <- `offsetX`,
`offset_y` <- `offsetY`
)
1 Like
3

Hi arnaud,
Thank you!

1 Like