Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-630

In DFSOutputStream.nextBlockOutputStream(), the client can exclude specific datanodes when locating the next block.

    Details

    • Hadoop Flags:
      Incompatible change, Reviewed

      Description

      created from hdfs-200.

      If during a write, the dfsclient sees that a block replica location for a newly allocated block is not-connectable, it re-requests the NN to get a fresh set of replica locations of the block. It tries this dfs.client.block.write.retries times (default 3), sleeping 6 seconds between each retry ( see DFSClient.nextBlockOutputStream).

      This setting works well when you have a reasonable size cluster; if u have few datanodes in the cluster, every retry maybe pick the dead-datanode and the above logic bails out.

      Our solution: when getting block location from namenode, we give nn the excluded datanodes. The list of dead datanodes is only for one block allocation.

        Attachments

        1. 0001-Fix-HDFS-630-0.21-svn.patch
          17 kB
          Cosmin Lehene
        2. 0001-Fix-HDFS-630-0.21-svn-1.patch
          17 kB
          Cosmin Lehene
        3. 0001-Fix-HDFS-630-0.21-svn-2.patch
          17 kB
          Cosmin Lehene
        4. 0001-Fix-HDFS-630-for-0.21.patch
          16 kB
          Cosmin Lehene
        5. 0001-Fix-HDFS-630-for-0.21-and-trunk-unified.patch
          18 kB
          Cosmin Lehene
        6. 0001-Fix-HDFS-630-svn.patch
          15 kB
          Cosmin Lehene
        7. 0001-Fix-HDFS-630-svn.patch
          15 kB
          Cosmin Lehene
        8. 0001-Fix-HDFS-630-trunk-svn-1.patch
          15 kB
          Cosmin Lehene
        9. 0001-Fix-HDFS-630-trunk-svn-2.patch
          15 kB
          Cosmin Lehene
        10. 0001-Fix-HDFS-630-trunk-svn-3.patch
          17 kB
          stack
        11. 0001-Fix-HDFS-630-trunk-svn-3.patch
          17 kB
          Cosmin Lehene
        12. 0001-Fix-HDFS-630-trunk-svn-4.patch
          17 kB
          Cosmin Lehene
        13. HDFS-630.20-security.1.patch
          14 kB
          Jitendra Nath Pandey
        14. HDFS-630.patch
          9 kB
          Cosmin Lehene
        15. hdfs-630-0.20.txt
          14 kB
          Todd Lipcon
        16. hdfs-630-0.20-append.patch
          13 kB
          Nicolas Spiegelberg

          Issue Links

            Activity

              People

              • Assignee:
                clehene Cosmin Lehene
                Reporter:
                mry.maillist Ruyue Ma
              • Votes:
                1 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: