Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8863

The remaining space check in BlockPlacementPolicyDefault is flawed

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      The block placement policy calls DatanodeDescriptor#getRemaining(StorageType}} to check whether the block is going to fit. Since the method is adding up all remaining spaces, namenode can allocate a new block on a full node. This causes pipeline construction failure and abandonBlock. If the cluster is nearly full, the client might hit this multiple times and the write can fail permanently.

      Attachments

        1. HDFS-8863-branch-2.6.1.txt
          10 kB
          Vinod Kumar Vavilapalli
        2. HDFS-8863.v3.patch
          9 kB
          Kihwal Lee
        3. HDFS-8863.v2.patch
          7 kB
          Kihwal Lee
        4. HDFS-8863.patch
          7 kB
          Kihwal Lee

        Issue Links

          Activity

            People

              kihwal Kihwal Lee
              kihwal Kihwal Lee
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: