[HUDI-6404] Create a new clustering strategy to execute parquet tools commands during clustering

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: clustering, table-service
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Surya Prasanna Yalla
• Created: 17/Jun/23 22:36
• Updated: 28/Mar/24 00:55

Description

Create a new clustering strategy to execute parquet tools commands during clustering.

If there is a use case of pruning some columns to save storage memory, current approach of clustering will iterate over every record and remove the unused column, this is so much time consuming. By directly using ParquetTools we can achieve this by running a command within the clustering strategy.

Here, the logic goes through the process of creating marker files that on event of failures we could use the existing rollback mechanism to remove the inflight files and parquet files.

Issue Links

• links to: GitHub Pull Request #9006