[HDFS-13677] Dynamic refresh Disk configuration results in overwriting VolumeMap

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3, 2.10.2
• Hadoop Flags: Reviewed
• Flags: Patch
• Assignee: ZanderXu
• Reporter: ZanderXu
• Created: 14/Jun/18 05:19
• Updated: 02/Mar/21 20:52
• Resolved: 29/Apr/19 23:25

Description

When I added a new disk by dynamically refreshing the configuration, an exception "FileNotFound while finding block" was caused.

 

The steps are as follows:

1.Change the hdfs-site.xml of DataNode to add a new disk.

2.Refresh the configuration by "./bin/hdfs dfsadmin -reconfig datanode ****:50020 start"

 

The error is like:

```

VolumeScannerThread(/media/disk5/hdfs/dn): FileNotFound while finding block BP-233501496-...-1514185698256:blk_1620868560_547245090 on volume /media/disk5/hdfs/dn

org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-1997955181-...-1514186468560:blk_1090885868_17145082
at org.apache.hadoop.hdfs.server.datanode.BlockSender.getReplica(BlockSender.java:471)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:240)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:553)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:148)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:254)
at java.lang.Thread.run(Thread.java:748)

```

I added some logs for confirmation, as follows:

Log Code like:

And the result is like:

 

The Size of 'VolumeMap' has been reduced, and We found the 'VolumeMap' to be overridden by the new Disk Block by the method 'ReplicaMap.addAll(ReplicaMap other)'.

 

Issue Links

• links to: GitHub Pull Request #780