[HUDI-3290] Make the .hoodie-partition-metadata as empty parquet file

• Type: New Feature
• Status: Closed
• Priority: Blocker
• Resolution: Done
• Fix Version/s: 0.11.0
• Component/s: metadata
• Labels: pull-request-available user-support-issues
• Story Points: 5
• Epic Link: Hudi-BigQuery
• Assignee: Prashant Wason
• Reporter: Vinoth Govindarajan

Reviewers

Shiyan Xu, sivabalan narayanan, Vinoth Chandar

• Created: 21/Jan/22 06:24
• Updated: 04/Apr/22 23:30
• Resolved: 04/Apr/22 15:21

Description

For BigQuery and Snowflake integration, we can't able to create external tables when the partition folder has a non-parquet file `.hoodie-partition-metadata`.

I understand this is an important file to find the .hoodie folder from within the partition folder, the long term solution is to get rid of this file, but as a short term solution if we can convert this to an empty parquet file and add the necessary depth information in the footer, then it will pass the BigQuery/Snowflake external table validation and allow us to create an external parquet table on top of hudi folder structure.

Issue Links

• is duplicated by: HUDI-3433 Migrate _hoodie_partition_metadata file to parquet to ensure bigquery is happy
• links to: GitHub Pull Request #5179