Hi
I have quite a lot of pipelines … hundreds like this one:
create pipeline pipeline_1_dc1
as load data kafka ‘kafka-0-some.where:9094,kafka-1-some.where.dc1:9094,kafka-2-some.where.dc1:9094,kafka-3-some.where.dc1:9094,kafka-4-some.where.dc1:9094,kafka-5-some.where.dc1:9094,kafka-6-some.where.dc1:9094,kafka-7-some.where.dc1:9094,kafka-8-some.where.dc1:9094/topic1’
batch_interval 1000
…
They all contains list of the brokers
There are no more 5 such lists
So I’m wondering if there is a way to create 5 links in order to avoid endless repeating kafka brokers in each pipeline:
create link kafkaCluster1 as kafka
config ‘{“hosts”: [“kafka-0-some.where:9094”,“kafka-1-some.where.dc1:9094”,“kafka-2-some.where.dc1:9094”,“kafka-3-some.where.dc1:9094”,“kafka-4-some.where.dc1:9094”,“kafka-5-some.where.dc1:9094”,“kafka-6-some.where.dc1:9094”,“kafka-7-some.where.dc1:9094”,“kafka-8-some.where.dc1:9094”]}’
credentials ‘{}’
;
create pipeline pipeline_1_dc1
as load data link kafkaCluster1 ‘topic1’
batch_interval 1000
…
create pipeline pipeline_2_dc1
as load data link kafkaCluster1 ‘topic2’
batch_interval 1000
…
so on
Regards,
Andrii