[HDFS-10563] Block reports could be silently dropped by NN

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Invalid
• Affects Version/s: 2.8.0, 2.7.3
• Component/s: namenode
• Assignee: Unassigned
• Reporter: Staffan Friberg
• Created: 22/Jun/16 18:59
• Updated: 23/Jun/16 14:28
• Resolved: 23/Jun/16 14:28

Description

Reading through the block reporting code I think I've spotted a case when block reports can silently be dropped and leave thread waiting indefinitely on a FutureTask that never will be executed.

The BlockReportProcessingThread.enqueue method doesn't return any status on if the enqueuing of the task was successful and does not handle the case when the queue is full and offer return false.

Going back through the call stack to BlockManager.runBlockOp, which indirectly calls enqueue with a FutureTask and then proceeds to do get() om the task.

So if the internal queue in the BlockReportingProcessingThread is full, the BR would never be handled and the thread queuing the task would wait indefinitely on the FutureTask that will never be executed.

Issue Links

• relates to: HDFS-9198 Coalesce IBR processing in the NN