[HDFS-8964] When validating the edit log, do not read at or beyond the file offset that is being written

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.1
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: journal-node, namenode
• Target Version/s: 2.8.0
• Assignee: Zhe Zhang
• Reporter: Zhe Zhang
• Created: 26/Aug/15 21:00
• Updated: 30/Aug/16 01:28
• Resolved: 03/Sep/15 18:25

Description

NN/JN validates in-progress edit log files in multiple scenarios, via EditLogFile#validateLog. The method scans through the edit log file to find the last transaction ID.

However, an in-progress edit log file could be actively written to, which creates a race condition and causes incorrect data to be read (and later we attempt to interpret the data as ops). This causes problems for INotify, which reads edit log entries while the edit log is still being written.

Currently validateLog is used in 3 places:

1. NN getEditsFromTxid
2. JN getEditLogManifest
3. NN/JN recoverUnfinalizedSegments

In the first two scenarios we should provide a maximum TxId to validate in the in-progress file. The 3rd scenario won't cause a race condition because only non-current in-progress edit log files are validated.

validateLog is actually only used with in-progress files, and could use a better name and Javadoc.