[HUDI-2780] Mor reads the log file and skips the complete block as a bad block, resulting in data loss

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.12.1
• Labels: core-flow-ds pull-request-available sev:critical
• Story Points: 0.5
• Assignee: jing
• Reporter: jing

Reviewers

Alexey Kudinkin, sivabalan narayanan

• Created: 17/Nov/21 07:46
• Updated: 28/Sep/22 15:34
• Resolved: 28/Sep/22 15:34

Description

Check the data in the middle of the bad block through debug, and find that the lost data is in the offset of the bad block, but because of the eof skip during the reading, the compact merge cannot be written to the parquet at that time, but the deltacommit of the time is successful. There are two consecutive hudi magic in the middle of the bad block. Reading blocksize in the next digit actually reads the binary conversion of #HUDI# to 1227030528, which means that the eof exception is reported when the file size is exceeded.

Detect the position of the next block and skip the bad block. It should not start from the position after reading the blocksize, but from the position before reading the blocksize

Issue Links

• links to: GitHub Pull Request #4015