[HDFS-7915] The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.7.0, 2.6.1, 3.0.0-alpha1
• Labels: 2.6.1-candidate
• Target Version/s: 2.7.0
• Assignee: Colin McCabe
• Reporter: Colin McCabe
• Created: 11/Mar/15 06:09
• Updated: 30/Jun/17 13:36
• Resolved: 15/Mar/15 08:42

Description

The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error. In DataXceiver#requestShortCircuitFds, the DataNode can succeed at the first part (mark the slot as used) and fail at the second part (tell the DFSClient what it did). The "try" block for unregistering the slot only covers a failure in the first part, not the second part. In this way, a divergence can form between the views of which slots are allocated on DFSClient and on server.

Issue Links

• breaks: HDFS-8070 Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode
• is broken by: HDFS-9466 TestShortCircuitCache#testDataXceiverCleansUpSlotsOnFailure is flaky
• is related to: HADOOP-11802 DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm