[HDFS-16752] 2.7.2 Rolling Upgrade 3.3.4 Datanode cannot be Degraded due to version Inconsistency

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.7.2, 3.3.4
• Component/s: datanode, rolling upgrades
• Assignee: Unassigned
• Reporter: Yanlei Yu
• Created: 30/Aug/22 07:27
• Updated: 05/Sep/22 09:08
• Resolved: 05/Sep/22 09:08

Description

I am following the official website: HDFS Rolling Upgrade（https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html）: After the Rolling Upgrade (HDFS dfsAdmin -RollingUpgrade Finalize is not executed), start degradeing Datanode . However, Datanode degradation fails. The log of Datanode is displayed as follows:

myhost001.datanode.log

INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data01/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data01/block. Reported: -57. Expecting = -56.
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data02/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data02/block. Reported: -57. Expecting = -56.
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data03/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data03/block. Reported: -57. Expecting = -56.
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data01/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data01/block. Reported: -57. Expecting = -56.
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data02/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data02/block. Reported: -57. Expecting = -56.
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data03/block/in_use.lock acquired by nodename 37422@myhost001
WARN org.apache.hadoop.hdfs.server.common.Storage: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data03/block. Reported: -57. Expecting = -56.
FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to myhost002/***:9002. Exiting.
java.io.IOException: All specified directories are failed to load.
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1393)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1358)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:644)
        at java.lang.Thread.run(Thread.java:748)
FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to myhost002/****:9002. Exiting.
java.io.IOException: All specified directories are failed to load.
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1393)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1358)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:644)
        at java.lang.Thread.run(Thread.java:748)
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to myhost002/****:9002
2022-08-30 15:09:04,908 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to myhost002/****:9002
2022-08-30 15:09:05,009 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool <registering> (Datanode Uuid unassigned)
2022-08-30 15:09:07,009 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2022-08-30 15:09:07,010 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2022-08-30 15:09:07,012 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at myhost001/****
************************************************************/