Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3672

Expose disk-location information for blocks to enable better scheduling

        Details

        • Type: Improvement Improvement
        • Status: Closed
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: 2.0.0-alpha
        • Fix Version/s: 2.0.2-alpha
        • Component/s: None
        • Labels:
          None
        • Hadoop Flags:
          Reviewed

          Description

          Currently, HDFS exposes on which datanodes a block resides, which allows clients to make scheduling decisions for locality and load balancing. Extending this to also expose on which disk on a datanode a block resides would enable even better scheduling, on a per-disk rather than coarse per-datanode basis.

          This API would likely look similar to Filesystem#getFileBlockLocations, but also involve a series of RPCs to the responsible datanodes to determine disk ids.

          1. Text File
            hdfs-3672-1.patch
            37 kB
            Andrew Wang
          2. Text File
            hdfs-3672-2.patch
            48 kB
            Andrew Wang
          3. Text File
            hdfs-3672-3.patch
            49 kB
            Andrew Wang
          4. Text File
            hdfs-3672-4.patch
            52 kB
            Andrew Wang
          5. PDF File
            design-doc-v1.pdf
            73 kB
            Andrew Wang
          6. Text File
            hdfs-3672-5.patch
            52 kB
            Andrew Wang
          7. Text File
            hdfs-3672-6.patch
            60 kB
            Andrew Wang
          8. Text File
            hdfs-3672-7.patch
            61 kB
            Andrew Wang
          9. Text File
            hdfs-3672-8.patch
            61 kB
            Andrew Wang
          10. PDF File
            design-doc-v2.pdf
            73 kB
            Andrew Wang
          11. Text File
            hdfs-3672-9.patch
            68 kB
            Andrew Wang
          12. Text File
            hdfs-3672-10.patch
            68 kB
            Andrew Wang
          13. Text File
            hdfs-3672-11.patch
            68 kB
            Andrew Wang
          14. Text File
            hdfs-3672-12.patch
            69 kB
            Andrew Wang

            Issue Links

            is depended upon by

            Brainstorming - A place to record back and forth on notions not yet formed enough to make a 'New Feature' or 'Task' HBASE-6572 Tiered HFile storage

            • Major - Major loss of function.
            • Open
            is related to

            Bug - A problem which impairs or prevents the functions of the product. HDFS-3969 Small bug fixes and improvements for disk locations API

            • Major - Major loss of function.
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-4577 HDFS-3672 broke TestCombineFileInputFormat.testMissingBlocks() test

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed

            New Feature - A new feature of the product, which has yet to be developed. HDFS-2832 Enable support for heterogeneous storages in HDFS - DN as a collection of storages

            • Major - Major loss of function.
            • Resolved

            Brainstorming - A place to record back and forth on notions not yet formed enough to make a 'New Feature' or 'Task' HBASE-6572 Tiered HFile storage

            • Major - Major loss of function.
            • Open

              Activity

                People

                • Assignee:
                  Andrew Wang
                  Reporter:
                  Andrew Wang
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  33 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development