Running into pipeline error with kafka ssl

Help Connectors and Pipelines
1

We are running into issue with Kafka pipeline. The pipeline works fine if it doesn’t need a certificate.

And running into error when the broker needs certificate.

Pipeline Command :-

CREATE PIPELINE caffeMac_appleMerchant AS LOAD DATA KAFKA ‘host:9095/test_topic’ CONFIG ‘{“security.protocol”: “SSL”, “ssl.certificate.location”: “/secrets/truststore.pem”, “ssl.key.location”: “/secrets/keystore.pem”, “ssl.ca.location”: “/secrets/ca.pem”}’ CREDENTIALS ‘{“ssl.key.password”: “XXXXX"}’ INTO PROCEDURE test_proc

Error:-

ERROR 1933 ER_EXTRACTOR_EXTRACTOR_GET_LATEST_OFFSETS: Cannot get source metadata for pipeline. Stderr: 2021-04-23 Batch starting with new consumer. 2021-04-23 21:29:00.006 Batch starting with new consumer.

We are able to consume with Java program.

properties.put(“bootstrap.servers”,“host:9095”);
properties.put(“key.deserializer”, “org.apache.kafka.common.serialization.StringDeserializer”);
properties.put(“value.deserializer”, “org.apache.kafka.common.serialization.StringDeserializer”);
properties.put(“group.id”, “test123”);
properties.put(“auto.offset.reset”, “earliest”);
properties.put(“security.protocol”, “SSL”);
properties.put(“ssl.truststore.location”, “/secrets/truststore.jks”);
properties.put(“ssl.truststore.password”, “XXXXXX");
properties.put(“ssl.keystore.location”, “/secrets/keystore.jks”);
properties.put(“ssl.keystore.password”, “XXXXXXX");
properties.put(“max.poll.records”, “50”);

For java program, we need,

ssl.truststore.location
ssl.keystore.location

The MemSQL pipeline needs,

ssl.certificate.location
ssl.key.location
ssl.ca.location

Here, we use per file for ssl.ceritifcate location without password.
For ssl.key.location, using key pen file.
For ssl.ca.location - using our root ca pen file.

Is there anything wrong with that?

2

Hello @udayabaski! Thanks for trying out Kafka SSL pipelines.

As you noticed, pipelines need a PEM formatted key, certificate, and CA, which can be extracted from truststore.jks and keystore.jks. It looks to me that you should have all the necessary configurations. Could you get more logs please?

SET GLOBAL pipelines_extractor_debug_logging=ON;
FLUSH EXTRACTOR POOLS;
CREATE PIPELINE caffeMac_appleMerchant AS LOAD DATA KAFKA ‘host:9095/test_topic’ CONFIG ‘{“security.protocol”: “SSL”, “ssl.certificate.location”: “/secrets/truststore.pem”, “ssl.key.location”: “/secrets/keystore.pem”, “ssl.ca.location”: “/secrets/ca.pem”}’ CREDENTIALS ‘{“ssl.key.password”: “XXXXX”}’;
SHOW WARNINGS;