Pipeline unable to ingest JSON data with missing fields

Help Connectors and Pipelines
1

Hi,
I am trying to ingest JSON data from Kafka but it’s failing since some messages do not have all the fields in it.

I have complete list of fields that may be part of JSON in some cases while in other cases it contains only subset of fields(subset could be 1 or more fields). How can I specify when a field doesn’t exist then use default value while mapping it to a table column?

Thanks!

2

I was able to solve it by specifying default clause and literal expr was assigned to the column when path can’t be found in input JSON

1 Like