Ingest parquet with pipelines multiple set clauses for datetime objects

1

I need to convert datetime objects in my pipeline unfortunately i cannot find any documentation on the syntax for multiple set clauses for multiple columns.
Example:
CREATE PIPELINE testpipeline AS LOAD DATA S3 “s3://test-bucket/modules/flow/snowflakeExportTemp/17c3e786-b376-4f4d-9ef5-2f30849f1f26/*” CONFIG ’
{
“region”: “eu-central-1”
}
’ CREDENTIALS ’
{
“aws_access_key_id”:“”,
“aws_secret_access_key”: “”,
“aws_session_token:”"
}
’ INTO TABLE TestTable FORMAT PARQUET (AMOUNT ← %::AMOUNT,@CREATEDATE ← %::CREATEDATE,CREATE DATE V1 ← %::CREATE DATE V1,test/column ← %::test/column) SET CREATEDATE = timestampadd(MICROSECOND, @CREATEDATE*1000, from_unixtime(0));

How to add the conversion for multiple date columns? Thanks

2

To convert multiple datetime objects in your pipeline, you can extend the SET clause used in your CREATE PIPELINE statement to include additional datetime columns. Assuming you have multiple date columns that you want to convert, you would repeat the pattern you have shown for the CREATEDATE column for each additional datetime column you wish to convert.

3

Thank you for sharing…

1 Like