Ingesting Partitioned Parquet data from pipeline

Help Connectors and Pipelines
1

Pipeline is not reading data from partitioned Parquet files

2

Hi kshastry, welcome to the SingleStore forums!

We’d need more information to try to help you with this. For starters, check that the pipeline is created, started, and that the system can read the data from the data source, and that the data file schemas and target table schemas match up.

3

Hi hanson,

The sample parquet structure is

Data.parquet/DATE=2020-12-25/part-00000-8e301b7c-9739-481f-8794-f4dbc2a8b008.c000.snappy.parquet

My table has column DATE. But since the parquet is partitioned on DATE, it is not present in the parquet file. And hence the data is not ingested.
The pipeline is created. The pipeline can read the data from the source.

4

Hi,
we also face the same issue.

we are also using version 7.8.8
Source structure is having the partitioned table on 3 columns.

we are hitting the below error when trying to run the pipeline.
Path has no DEFAULT clause and was not found as a primitive field of input schema message hive_schema