Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13677

Dynamic refresh Disk configuration results in overwriting VolumeMap

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.10.0, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Flags:
      Patch

      Description

      When I added a new disk by dynamically refreshing the configuration, an exception "FileNotFound while finding block" was caused.

       

      The steps are as follows:

      1.Change the hdfs-site.xml of DataNode to add a new disk.

      2.Refresh the configuration by "./bin/hdfs dfsadmin -reconfig datanode ****:50020 start"

       

      The error is like:

      ```

      VolumeScannerThread(/media/disk5/hdfs/dn): FileNotFound while finding block BP-233501496-...-1514185698256:blk_1620868560_547245090 on volume /media/disk5/hdfs/dn

      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-1997955181-...-1514186468560:blk_1090885868_17145082
      at org.apache.hadoop.hdfs.server.datanode.BlockSender.getReplica(BlockSender.java:471)
      at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:240)
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:553)
      at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:148)
      at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103)
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:254)
      at java.lang.Thread.run(Thread.java:748)

      ```

      I added some logs for confirmation, as follows:

      Log Code like:

      And the result is like:

       

      The Size of 'VolumeMap' has been reduced, and We found the 'VolumeMap' to be overridden by the new Disk Block by the method 'ReplicaMap.addAll(ReplicaMap other)'.

       

        Attachments

        1. HDFS-13677-002-2.9-branch.patch
          7 kB
          Stephen O'Donnell
        2. HDFS-13677-002.patch
          7 kB
          xuzq
        3. HDFS-13677-001.patch
          4 kB
          xuzq
        4. image-2018-06-14-13-10-24-032.png
          465 kB
          xuzq
        5. image-2018-06-14-13-05-54-354.png
          164 kB
          xuzq

          Issue Links

            Activity

              People

              • Assignee:
                xuzq_zander xuzq
                Reporter:
                xuzq_zander xuzq
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: