Alter pipeline set offset does not work when loading data into proecdure

Help Connectors and Pipelines
1

I have Kafka pipeline which loads data using procedure. I wanted to set pipeline to latest Kafka offset but it gave an error.

2

could you post the SQL you tried to run and the error?

3

Can set pipeline to latest or earliest but not to a specific offset.

Pipeline is calling a stored procedure and is treating input message as string.
Signature:
CREATE PROCEDURE SP_LOAD_DATA(batch query(KAFKADATA text )) RETURNS void AS

Error Code: 1706. Feature ‘ALTER PIPELINE SET OFFSETS requires a JSON object containing string partition keys to integer offset values’ is not supported by MemSQL.

4

I see. we have an example in the documentation how to set pipeline offsets.

The JSON object it refers to is the following form:

memsql> ALTER PIPELINE mypipeline SET OFFSETS '{"0":100}, {"1":100}';

The JSON describes a map of kafka partitions to last ingested offset, so that pipeline can start from that point on.

5

Yes, had followed the documentation and tried the Alter.
ALTER PIPELINE MY_PIPELINE SET OFFSETS ‘{“0”:30000}, {“1”:30000}’;

PL offsets:
partition earliest latest
|0| 29842 |37338|
|1| 29839 |37336|

6

my apologies for misunderstanding your question. let me try to repro the error. what version of memsql are you using?

7

V6.8 Is the version of memsql

8

Just to update, it worked with following command.

ALTER PIPELINE MY_PIPELINE SET OFFSETS ‘{“0”:30000}’;
ALTER PIPELINE MY_PIPELINE SET OFFSETS ‘{“1”:30000}’;

9

oh, it’s JSON error in our docs. thanks for pointing it out. what you want is

ALTER PIPELINE mypipeline SET OFFSETS '{"0":100,"1":100}';