[HUDI-846] Turn on incremental cleaning bu default in 0.6.0

• Type: Sub-task
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.6.0, 0.5.3
• Component/s: cleaning
• Labels: pull-request-available
• Assignee: Balaji Varadarajan
• Reporter: Balaji Varadarajan
• Created: 29/Apr/20 19:30
• Updated: 23/May/20 12:29
• Resolved: 20/May/20 16:43

Description

Incremental cleaner will track commits that have happened since the last clean operation to figure out partitions which needs to be scanned for cleaning. This avoids the costly scanning of all partition paths.

Incremental cleaning is currently disabled by default. We need to enable it by default in 0.6.0.

No special handling is required for upgrade/downgrade scenarios as incremental cleaning relies on standard format of commit metadata 

Issue Links

• links to: GitHub Pull Request #1634