Append data to existing parquet file

[sashk8]

Hi,

Would it be possible to add a feature to append the data to an existing parquet file?

Use-case is the following: We are using odbc2parquet to export CDC (Change Data Capture) data from an SQL Server database. The data is being exported every few minutes, so we get a large amount of small files (~150 files per table per day, each file ~50kB) that are uploaded to a data lake and ingested to a data warehousing platform (Snowflake). The platform is optimized to ingest optimally the data file sizes between 100-200MB. lots of small files create additional costs/overhead and the performance is not optimal.

Best regards,
Sašo

[pacman82]

Hello Sašo,

a "true" append is not possible, or sensible. For once parquet writes a footer at the end of each file, and also the row groups are column based, and should probably be larger than 50k for most usecases.

Append is one thing, but merging another entirely. There are many ways to merge multiple parquet files with similar schemas into one big one. Regarding the scope of odbc2parquet it just aims to be binding between a relational database and parquet. Currently I would consider merging parquet files out of its scope.

One of the ways I found with a search to merge parquet files is utilizing parquet-tools (https://linuxcommandlibrary.com/man/parquet-tools). There are others, too. Hope this helps!

Cheers, Markus

[sashk8]

Hi Markus,

Thanks for the explaination and quick answer.

I don't know the details of parquet file format, so I assumed an append would be easy to do.

I'll look into other options then. I guess we'll probably have to make a separate job that will run on a time schedule and merge the small parquet files into a larger one.

Best regards,
Sašo

[pacman82]

Hello Sašo,

thanks for understanding. I don't get why the same job wouldn't be able to execute to command line tools one after the other? Anyhow, every solution would "concatinate" files by actually merging smaller ones with each other.

Cheers, Markus