[HDFS-13924] Handle BlockMissingException when reading from observer

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: HDFS-12943, 3.3.0
• Target Version/s: HDFS-12943
• Hadoop Flags: Reviewed
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 17/Sep/18 21:28
• Updated: 24/Dec/18 18:22
• Resolved: 24/Oct/18 05:53

Description

Internally we found that reading from ObserverNode may result to BlockMissingException. This may happen when the observer sees a smaller number of DNs than active (maybe due to communication issue with those DNs), or (we guess) late block reports from some DNs to the observer. This error happens in DFSInputStream#chooseDataNode, when no valid DN can be found for the LocatedBlock got from the NN side.

One potential solution (although a little hacky) is to ask the DFSInputStream to retry active when this happens. The retry logic already present in the code - we just have to dynamically set a flag to ask the ObserverReadProxyProvider try active in this case.

cc shv, xkrogen, vagarychen, zero45 for discussion.