Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3672

Expose disk-location information for blocks to enable better scheduling

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, HDFS exposes on which datanodes a block resides, which allows clients to make scheduling decisions for locality and load balancing. Extending this to also expose on which disk on a datanode a block resides would enable even better scheduling, on a per-disk rather than coarse per-datanode basis.

      This API would likely look similar to Filesystem#getFileBlockLocations, but also involve a series of RPCs to the responsible datanodes to determine disk ids.

      1. hdfs-3672-1.patch
        37 kB
        Andrew Wang
      2. hdfs-3672-2.patch
        48 kB
        Andrew Wang
      3. hdfs-3672-3.patch
        49 kB
        Andrew Wang
      4. hdfs-3672-4.patch
        52 kB
        Andrew Wang
      5. design-doc-v1.pdf
        73 kB
        Andrew Wang
      6. hdfs-3672-5.patch
        52 kB
        Andrew Wang
      7. hdfs-3672-6.patch
        60 kB
        Andrew Wang
      8. hdfs-3672-7.patch
        61 kB
        Andrew Wang
      9. hdfs-3672-8.patch
        61 kB
        Andrew Wang
      10. design-doc-v2.pdf
        73 kB
        Andrew Wang
      11. hdfs-3672-9.patch
        68 kB
        Andrew Wang
      12. hdfs-3672-10.patch
        68 kB
        Andrew Wang
      13. hdfs-3672-11.patch
        68 kB
        Andrew Wang
      14. hdfs-3672-12.patch
        69 kB
        Andrew Wang

        Issue Links

          Activity

            People

            • Assignee:
              Andrew Wang
              Reporter:
              Andrew Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              33 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development