Hi Team,
I’m having difficulty setting up a pipeline in SingleStore to use a Python script for data transformation. The script is hosted externally and is supposed to convert words to uppercase.
Here’s the relevant Python code hosted at http://abc.xyz/uppercase_transform.py:
#!/usr/bin/env python
import sys
for line in sys.stdin:
processed_line = line.upper()
sys.stdout.write(processed_line)
I created a table as follows:
CREATE TABLE words (
word VARCHAR(40)
);
However, my pipeline setup isn’t working:
CREATE OR REPLACE PIPELINE uppercase
LOAD DATA LINK azurelink ‘poc/words/’
BATCH_INTERVAL 3600000
DISABLE OFFSETS METADATA GC
WITH TRANSFORM (‘http://abc.xyz/uppercase_transform.py’, ‘’, ‘’)
INTO TABLE words
FIELDS TERMINATED BY ‘,’;
I have superuser access. Any assistance on getting this pipeline to function properly would be greatly appreciated!
Note: I want to keep my python code outside of singlestore
Thanks!