[HDFS-7725] Incorrect "nodes in service" metrics caused all writes to fail

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 2.7.2, 3.0.0-alpha1
• Labels: 2.7.2-candidate
• Assignee: Ming Ma
• Reporter: Ming Ma
• Created: 02/Feb/15 18:43
• Updated: 04/Jan/21 19:41
• Resolved: 08/Apr/15 23:02

Description

One of our clusters sometimes couldn't allocate blocks from any DNs. BlockPlacementPolicyDefault complains with the following messages for all DNs.

the node is too busy (load:x > y)

It turns out the HeartbeatManager's nodesInService was computed incorrectly when admins decomm or recomm dead nodes. Here are two scenarios.

• Decomm dead nodes. It turns out HDFS-7374 has fixed it; not sure if it is intentional. cc / Zhe Zhang, Andrew Wang, Aaron Myers Here is the sequence of event without HDFS-7374.
  • Cluster has one live node. nodesInService == 1
  • The node becomes dead. nodesInService == 0
  • Decomm the node. nodesInService == -1

• However, HDFS-7374 introduces another inconsistency when recomm is involved.
  • Cluster has one live node. nodesInService == 1
  • The node becomes dead. nodesInService == 0
  • Decomm the node. nodesInService == 0
  • Recomm the node. nodesInService == 1

Issue Links

• is duplicated by: HDFS-9287 Block placement completely fails if too many nodes are decommissioning
• is related to: HDFS-15761 Dead NORMAL DN shouldn't transit to DECOMMISSIONED immediately