[HDFS-15761] Dead NORMAL DN shouldn't transit to DECOMMISSIONED immediately

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Labels: pull-request-available
• Assignee: Ye Ni
• Reporter: Ye Ni
• Created: 04/Jan/21 19:29
• Updated: 05/Feb/21 09:46

Description

To decommission a dead DN, the complete logic should be
Dead, NORMAL -> Dead, DECOMMISSION_INPROGRESS -> Dead, DECOMMISSIONED

Currently logic:

If a DN is already dead when DECOMMISSIONING starts, it becomes DECOMMISSIONED immediately. DECOMMISSION_INPROGRESS is skipped.

This logic is introduced by HDFS-7374 which is made because of HDFS-6791.

HDFS-6791 keeps the node in DECOMMISSION_INPROGRESS state if the node becomes dead during decommission, which could possibly make a dead DN in DECOMMISSION_INPROGRESS forever, if the DN could never be alive.

However, putting a dead DN to DECOMMISSIONED directly is not secure. For example, 3 DN of the same block are dead at the same time, then the administrator wants to decommission them unintentionally. Namenode should check first before transit them to DECOMMISSIONED. Otherwise, it would be a data loss.

In this case, all 3 DNs can't become DECOMMISSIONED which is by design. The administrator needs to do some manual intervention, either repair the dead machine or service or recover the data before take action on them.

This change is to add Dead, DECOMMISSION_INPROGRESS back.
1. Dead normal DN is in DECOMMISSION_INPROGRESS first.
2. NN checks pendingReplicationBlocksCount and underReplicatedBlocksCount are both 0.
3. Transit the dead DN to DECOMMISSIONED.

2 is implemented by HDFS-7409, which adds a check to allow dead nodes in DECOMMISSION_IN_PROGRESS to progress to DECOMMISSIONED state if all files on the filesystem are fully-replicated.

Issue Links

• is caused by: HDFS-7374 Allow decommissioning of dead DataNodes
• relates to: HDFS-6791 A block could remain under replicated if all of its replicas are on decommissioned nodes
• relates to: HDFS-7725 Incorrect "nodes in service" metrics caused all writes to fail
• requires: HDFS-7409 Allow dead nodes to finish decommissioning if all files are fully replicated
• links to: GitHub Pull Request #2588