[HUDI-535] Compatibility Issue : Hudi Readers using newer layout version can see intermittent errors due to race conditions with concurrent compactions in MOR tables

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.5.1
• Component/s: Common Core
• Labels: pull-request-available
• Assignee: Balaji Varadarajan
• Reporter: Balaji Varadarajan
• Created: 14/Jan/20 18:07
• Updated: 03/Feb/20 03:16
• Resolved: 17/Jan/20 22:53

Description

As part of recent changes in 0.5.1 to avoid renames in metadata, the compaction plans are no longer getting written to .aux folder. Instead the upgraded reader reads from timeline (.hoodie). This would work as long as writer is also upgraded but if reader alone is upgraded, there would be conditions when hudi reader tried to read from compaction requested file which was renamed by concurrent compactions.

Issue Links

• links to: GitHub Pull Request #1229