[HUDI-2364] Run compaction without user schema file provided

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Labels: core-flow-ds pull-request-available sev:normal
• Assignee: Unassigned
• Reporter: Wenning Ding
• Created: 25/Aug/21 23:23
• Updated: 03/Mar/22 15:51
• Resolved: 03/Mar/22 15:51

Description

Currently to run Hudi compaction manually, customers have to pass the avsc file of data schema by themselves,
e.g. in Hudi CLI,

 

{{}}

compaction run --compactionInstant 20201203005420 \ --parallelism 2 --sparkMemory 2G \ --schemaFilePath s3://xxx/hudi/mor_schema.avsc \ --propsFilePath file:///home/hadoop/config.properties --retry 1

Let customers provide avsc file is not a good option. Some customers don’t know how to generate this schema file, and some customers pass the wrong schema file and get other exceptions. We should handle this logic inside Hudi if possible.

Issue Links

• duplicates: HUDI-3369 New ScheduleAndExecute mode for HoodieCompactor and hudi-cli
• links to: GitHub Pull Request #3540