Kafka Transform in Golang

Help Connectors and Pipelines
1

Despite my best efforts, my pipeline test always comes back with an error:

> TEST PIPELINE ApToLocationPipeline LIMIT 8; ERROR 1953 ER_TRANSFORM_NONZERO_EXIT_VALUE: Leaf Error (127.0.0.1:3306): Leaf Error (10.16.208.220:3306): Transform for pipeline ApToLocationPipeline exited with nonzero exit code. Stderr: 2019-08-01 19:33:55.523 ERROR: Subprocess /var/lib/memsql/33ec93d1-de16-4a5d-bd5f-03cfd30ee2d0/data/./transforms/f65997ec64701fb6306238ae6f250181d6d91041/5a7e05eade9e781496047e1390f126b2431f48fd/17ee93407e6bc984ad623c388063663a461fcc4f/5fc84cfa42e3a48fccb7faa78ff676a095d4c071_transform exited with failure result (8 : Exec format error)

I believe that this indicates I’ve not compiled a binary in a format that is compatible with the OS/Arch being used in this case. I’ve got my cluster set up using AWS CloudFormation, so the AMI is memsql-cloudformation_6.8.4-696e55c4e4_1.6.3_1.1.1 20190722-203114 (ami-0ecc473841aeb95f1) which is a standard linux, I’m running these on m4.2xlarge instances, which are of the x86 family.

I’ve tried compiling with GOARCH=amd64 and GOARCH=386 but I’m getting nowhere. Any ideas?

2

What’s the output of show create pipeline ApToLocationPipeline?

If you have access to that leaf node, you can rule out a genuine “not executable” error by copying the binary to the leaf node and running it manually. But I’d assume that the issue is instead some bad interaction with the TRANSFORM syntax.

I’ve also personally seen exec format error errors when attempting to execute shell scripts without a shebang as if they were binaries. Do you happen to have such a wrapper script around the binary?

3

Output:

show create pipeline ApToLocationPipeline
+-------------------------------------------------------------+
| Pipeline                     | Create Pipeline              |
+-------------------------------------------------------------+
| ApToLocationPipeline         | CREATE PIPELINE `ApToLocat...|
+-------------------------------------------------------------+
1 row in set (135 ms)

No wrapper. I uploaded the raw binary executable to use.

4

Update: Looked at the contents of the file that should be my executable, and lo and behold:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>9FE9A4E3404CDBD0</RequestId><HostId>NXLswDS0vyRQzpwq+gZzYFQTJYSdMcOnIXzn4V4Jlt9w6qPNcrWutqqM4B5/JNbFKSmeJD+sako=</HostId></Error>

So it’s not the executable it should be. Since I’m fetching from S3, I’m going to guess I have a permission problem, but seems odd MemSQL failed to mention that.

5

Yep, that was it. I set it to public read and problem solved.

1 Like