Modify AWS credentials for S3 pipeline

Help Connectors and Pipelines
1

We have a corporate policy to update our AWS credentials periodically. As such we need to modify the credentials on our memsql S3 ingestion pipelines.

I looked at the ALTER PIPELINE docs here:

It doesn’t appear that I can alter the CREDENTIALS field.
I have tried various ALTER statements with no success.

Is my only option to do a SHOW CREATE PIPELINE … EXTENDED, then STOP the pipeline. Then DESTROY the pipeline, then CREATE the pipeline using the output of SHOW CREATE… plus the modified AWS key?

2

Hi Ian

Thanks for posting a question

You can just do the following (though you do need to know the entire Pipelines statement, which you can know by doing SHOW PIPELINE my_pipeline):

CREATE OR REPLACE PIPELINE my_pipeline AS LOAD DATA S3 '/../../'
CREDENTIALS '{}'..
3

I ended up going through the process I suggested above. show create pipeline <my_pipeline> extended and saved to a file. Then I stopped the pipeline, destroyed the pipeline, then created the pipeline based on what I saved in the file with the updated credentials.

This process ensured I didn’t reprocess the files in the S3 bucket.

The actual process was slightly more involved due to special characters and the like. But some quick work in vi fixed most of that. I had to convert all “\n” strings to an actual \n character. Then I converted all the escaped values to non-escaped values.

Ideally I would have been able to do:

ALTER PIPELINE <my_pipeline> SET CREDENTIALS ‘{… json configuration with security information…}’;

I will likely build some scripts to do this automatically for our next rolling key changes.