Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3672

Expose disk-location information for blocks to enable better scheduling

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, HDFS exposes on which datanodes a block resides, which allows clients to make scheduling decisions for locality and load balancing. Extending this to also expose on which disk on a datanode a block resides would enable even better scheduling, on a per-disk rather than coarse per-datanode basis.

      This API would likely look similar to Filesystem#getFileBlockLocations, but also involve a series of RPCs to the responsible datanodes to determine disk ids.

      1. hdfs-3672-9.patch
        68 kB
        Andrew Wang
      2. hdfs-3672-8.patch
        61 kB
        Andrew Wang
      3. hdfs-3672-7.patch
        61 kB
        Andrew Wang
      4. hdfs-3672-6.patch
        60 kB
        Andrew Wang
      5. hdfs-3672-5.patch
        52 kB
        Andrew Wang
      6. hdfs-3672-4.patch
        52 kB
        Andrew Wang
      7. hdfs-3672-3.patch
        49 kB
        Andrew Wang
      8. hdfs-3672-2.patch
        48 kB
        Andrew Wang
      9. hdfs-3672-12.patch
        69 kB
        Andrew Wang
      10. hdfs-3672-11.patch
        68 kB
        Andrew Wang
      11. hdfs-3672-10.patch
        68 kB
        Andrew Wang
      12. hdfs-3672-1.patch
        37 kB
        Andrew Wang
      13. design-doc-v2.pdf
        73 kB
        Andrew Wang
      14. design-doc-v1.pdf
        73 kB
        Andrew Wang

        Issue Links

          Activity

          Andrew Wang created issue -
          Andrew Wang made changes -
          Field Original Value New Value
          Attachment hdfs-3672-1.patch [ 12537292 ]
          Andrew Wang made changes -
          Attachment hdfs-3672-2.patch [ 12537937 ]
          Andrew Wang made changes -
          Attachment hdfs-3672-3.patch [ 12538086 ]
          Andrew Wang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jeff Hammerbacher made changes -
          Link This issue is related to HDFS-2832 [ HDFS-2832 ]
          Andrew Wang made changes -
          Attachment hdfs-3672-4.patch [ 12538617 ]
          Andrew Wang made changes -
          Attachment design-doc-v1.pdf [ 12538854 ]
          Andrew Wang made changes -
          Attachment hdfs-3672-5.patch [ 12539102 ]
          Andrew Wang made changes -
          Attachment hdfs-3672-6.patch [ 12539391 ]
          Andrew Wang made changes -
          Attachment hdfs-3672-7.patch [ 12539673 ]
          Andrew Wang made changes -
          Attachment hdfs-3672-8.patch [ 12539712 ]
          Andrew Wang made changes -
          Attachment design-doc-v2.pdf [ 12539964 ]
          Jeff Hammerbacher made changes -
          Link This issue is related to HBASE-6572 [ HBASE-6572 ]
          Andrew Wang made changes -
          Attachment hdfs-3672-9.patch [ 12540815 ]
          Andrew Wang made changes -
          Link This issue is depended upon by HBASE-6572 [ HBASE-6572 ]
          Andrew Wang made changes -
          Attachment hdfs-3672-10.patch [ 12541273 ]
          Andrew Wang made changes -
          Attachment hdfs-3672-11.patch [ 12541275 ]
          Andrew Wang made changes -
          Attachment hdfs-3672-12.patch [ 12541314 ]
          Aaron T. Myers made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.2.0-alpha [ 12322472 ]
          Resolution Fixed [ 1 ]
          Alejandro Abdelnur made changes -
          Link This issue is related to HDFS-3842 [ HDFS-3842 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Andrew Wang made changes -
          Link This issue is related to HDFS-3969 [ HDFS-3969 ]

            People

            • Assignee:
              Andrew Wang
              Reporter:
              Andrew Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              33 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development