Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13677

Dynamic refresh Disk configuration results in overwriting VolumeMap

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3, 2.10.2
    • None
    • None
    • Reviewed
    • Patch

    Description

      When I added a new disk by dynamically refreshing the configuration, an exception "FileNotFound while finding block" was caused.

       

      The steps are as follows:

      1.Change the hdfs-site.xml of DataNode to add a new disk.

      2.Refresh the configuration by "./bin/hdfs dfsadmin -reconfig datanode ****:50020 start"

       

      The error is like:

      ```

      VolumeScannerThread(/media/disk5/hdfs/dn): FileNotFound while finding block BP-233501496-...-1514185698256:blk_1620868560_547245090 on volume /media/disk5/hdfs/dn

      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-1997955181-...-1514186468560:blk_1090885868_17145082
      at org.apache.hadoop.hdfs.server.datanode.BlockSender.getReplica(BlockSender.java:471)
      at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:240)
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:553)
      at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:148)
      at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103)
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:254)
      at java.lang.Thread.run(Thread.java:748)

      ```

      I added some logs for confirmation, as follows:

      Log Code like:

      And the result is like:

       

      The Size of 'VolumeMap' has been reduced, and We found the 'VolumeMap' to be overridden by the new Disk Block by the method 'ReplicaMap.addAll(ReplicaMap other)'.

       

      Attachments

        1. HDFS-13677-001.patch
          4 kB
          ZanderXu
        2. HDFS-13677-002.patch
          7 kB
          ZanderXu
        3. HDFS-13677-002-2.9-branch.patch
          7 kB
          Stephen O'Donnell
        4. image-2018-06-14-13-05-54-354.png
          164 kB
          ZanderXu
        5. image-2018-06-14-13-10-24-032.png
          465 kB
          ZanderXu

        Issue Links

          Activity

            People

              xuzq_zander ZanderXu
              xuzq_zander ZanderXu
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: