Java Pipeline Transform - Transform Error

Help
1

To understand how the Pipeline transform works, I’m trying to execute a Java program through a shell script. But getting the below error. Need guidance to identify the possible cause as I was not able to debug it much further.

SinglestoreDB Version : 7.8.8
Environment : Docker Desktop

Pipeline Creation Statement:
CREATE PIPELINE IF NOT EXISTS dummy_event_pipeline
AS LOAD DATA S3 ‘dummy’
CONFIG ‘{“region”: “us-east-1”, “endpoint_url”:“http://172.17.0.2:9000”}’
CREDENTIALS ‘{“aws_access_key_id”:“yyyyyy”,“aws_secret_access_key”: “xxxx”}’
BATCH_INTERVAL 10
WITH TRANSFORM(‘http://127.0.0.1:8080/bin/run.sh’,‘’,'’)
INTO TABLE DUMMY_EVENTS;

Script

#!/bin/sh
java -cp /eclipse-workspace/singlestore/target/sample-singlestore-0.1.jar:. com.test.singlestore.App

Error information from Information_schema.PIPELINE_ERRORS

Transform for pipeline exited with nonzero exit code. Stderr: 2022-08-24 16:11:28.388 ERROR: Subprocess /var/lib/memsql/ff226ad9-46fd-4662-ba70-8c5c27cefb5d/data/./transforms/a94a8fe5ccb19ba61c4c0873d391e987982fbbd3/cf2a4c74ee3e7366a7535df123aabb6c88aabf36/c960c96d350d916372fa60d38fa84d300524de4f/5aade35bf71632659201571df726877e4e5046e0_transform exited with failure result (8 : Exec format error)

2

Sorry you are experiencing this error, Subash! Thanks for including the version you’re running. That’s helpful! I’ve passed this on to the internal support team for further guidance. We appreciate your patience. :pray:

3

Thanks Maria.

Error information from Information_schema.PIPELINE_ERRORS

Transform for pipeline exited with nonzero exit code. Stderr: 2022-08-24 16:11:28.388 ERROR: Subprocess /var/lib/memsql/ff226ad9-46fd-4662-ba70-8c5c27cefb5d/data/./transforms/a94a8fe5ccb19ba61c4c0873d391e987982fbbd3/cf2a4c74ee3e7366a7535df123aabb6c88aabf36/c960c96d350d916372fa60d38fa84d300524de4f/5aade35bf71632659201571df726877e4e5046e0_transform exited with failure result (8 : Exec format error)

4

We execute the provided transform script with exec(), which has slightly different semantics than what you’ll see just testing it from the command line. In particular, if it’s a script (python, bash, etc.) rather than an ELF executable, it must start with a shebang. That error is expected when there either isn’t a shebang, or else the shebang references an interpreter binary that doesn’t actually exist at that path on the SingleStore node running the transform (best bet is to just ensure that it exists on every node).

We mention that error here:

5

As we are running Singlestore on Docker, how do we ensure that interpreter binary is available inside the container. My assumption is that it is already preloaded into the container image. Isn’t that the case?

6

Do you start your python script with this #! (Shebang)?

7

Yes, I have already pasted my shell script in my question. It has the required shebang

8

Debugging steps:

  • Confirm that the script works inside the docker container by execing it directly (i.e. ./your_script.sh)
  • Confirm that /eclipse-workspace/singlestore/target/sample-singlestore-0.1.jar is mounted in your container
  • Try adding the script to the container directly (i.e. via a docker mount) and exec’ing it via absolute path rather than serving it over http
  • Try switching the WITH TRANSFORM call to exec java directly: WITH TRANSFORM('file://path/to/java', '', '-cp /eclipse-workspace/singlestore/target/sample-singlestore-0.1.jar:. com.test.singlestore.App')
1 Like
9

Also, confirm that java is actually available in the container.

1 Like