Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-630

In DFSOutputStream.nextBlockOutputStream(), the client can exclude specific datanodes when locating the next block.

    Details

    • Hadoop Flags:
      Incompatible change, Reviewed

      Description

      created from hdfs-200.

      If during a write, the dfsclient sees that a block replica location for a newly allocated block is not-connectable, it re-requests the NN to get a fresh set of replica locations of the block. It tries this dfs.client.block.write.retries times (default 3), sleeping 6 seconds between each retry ( see DFSClient.nextBlockOutputStream).

      This setting works well when you have a reasonable size cluster; if u have few datanodes in the cluster, every retry maybe pick the dead-datanode and the above logic bails out.

      Our solution: when getting block location from namenode, we give nn the excluded datanodes. The list of dead datanodes is only for one block allocation.

      1. 0001-Fix-HDFS-630-0.21-svn.patch
        17 kB
        Cosmin Lehene
      2. 0001-Fix-HDFS-630-0.21-svn-1.patch
        17 kB
        Cosmin Lehene
      3. 0001-Fix-HDFS-630-0.21-svn-2.patch
        17 kB
        Cosmin Lehene
      4. 0001-Fix-HDFS-630-for-0.21.patch
        16 kB
        Cosmin Lehene
      5. 0001-Fix-HDFS-630-for-0.21-and-trunk-unified.patch
        18 kB
        Cosmin Lehene
      6. 0001-Fix-HDFS-630-svn.patch
        15 kB
        Cosmin Lehene
      7. 0001-Fix-HDFS-630-svn.patch
        15 kB
        Cosmin Lehene
      8. 0001-Fix-HDFS-630-trunk-svn-1.patch
        15 kB
        Cosmin Lehene
      9. 0001-Fix-HDFS-630-trunk-svn-2.patch
        15 kB
        Cosmin Lehene
      10. 0001-Fix-HDFS-630-trunk-svn-3.patch
        17 kB
        stack
      11. 0001-Fix-HDFS-630-trunk-svn-3.patch
        17 kB
        Cosmin Lehene
      12. 0001-Fix-HDFS-630-trunk-svn-4.patch
        17 kB
        Cosmin Lehene
      13. HDFS-630.20-security.1.patch
        14 kB
        Jitendra Nath Pandey
      14. HDFS-630.patch
        9 kB
        Cosmin Lehene
      15. hdfs-630-0.20.txt
        14 kB
        Todd Lipcon
      16. hdfs-630-0.20-append.patch
        13 kB
        Nicolas Spiegelberg

        Issue Links

          Activity

            People

            • Assignee:
              Cosmin Lehene
              Reporter:
              Ruyue Ma
            • Votes:
              1 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development