i am creating pipeline taking parquet format from GCS. Currently, our parquet schema has 3 columns (a1,a2,a3). Now in future, one extra column will come in parquet. I wanted to make pipeline beforehand so that when that column comes in, I get the actual value. But when i change the parquet format and add that future column, pipeline gives error that column doesn’t exist in parquet schema.
Is there any way, i can specify unknown (future) column in advance for parquet and if it doesn’t comes in, set to default value or NULL?