Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3672

Expose disk-location information for blocks to enable better scheduling

          Details

          • Type: Improvement
          • Status: Closed
          • Priority: Major
          • Resolution: Fixed
          • Affects Version/s: 2.0.0-alpha
          • Fix Version/s: 2.0.2-alpha
          • Component/s: None
          • Labels:
            None
          • Hadoop Flags:
            Reviewed

            Description

            Currently, HDFS exposes on which datanodes a block resides, which allows clients to make scheduling decisions for locality and load balancing. Extending this to also expose on which disk on a datanode a block resides would enable even better scheduling, on a per-disk rather than coarse per-datanode basis.

            This API would likely look similar to Filesystem#getFileBlockLocations, but also involve a series of RPCs to the responsible datanodes to determine disk ids.

              Attachments

              1. design-doc-v1.pdf
                73 kB
                Andrew Wang
              2. design-doc-v2.pdf
                73 kB
                Andrew Wang
              3. hdfs-3672-1.patch
                37 kB
                Andrew Wang
              4. hdfs-3672-10.patch
                68 kB
                Andrew Wang
              5. hdfs-3672-11.patch
                68 kB
                Andrew Wang
              6. hdfs-3672-12.patch
                69 kB
                Andrew Wang
              7. hdfs-3672-2.patch
                48 kB
                Andrew Wang
              8. hdfs-3672-3.patch
                49 kB
                Andrew Wang
              9. hdfs-3672-4.patch
                52 kB
                Andrew Wang
              10. hdfs-3672-5.patch
                52 kB
                Andrew Wang
              11. hdfs-3672-6.patch
                60 kB
                Andrew Wang
              12. hdfs-3672-7.patch
                61 kB
                Andrew Wang
              13. hdfs-3672-8.patch
                61 kB
                Andrew Wang
              14. hdfs-3672-9.patch
                68 kB
                Andrew Wang

                Issue Links

                is depended upon by

                Brainstorming - A place to record back and forth on notions not yet formed enough to make a 'New Feature' or 'Task' HBASE-6572 Tiered HFile storage

                • Major - Major loss of function.
                • Open
                is related to

                Bug - A problem which impairs or prevents the functions of the product. HDFS-3969 Small bug fixes and improvements for disk locations API

                • Major - Major loss of function.
                • Closed

                Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-4577 HDFS-3672 broke TestCombineFileInputFormat.testMissingBlocks() test

                • Minor - Minor loss of function, or other problem where easy workaround is present.
                • Closed

                New Feature - A new feature of the product, which has yet to be developed. HDFS-2832 Enable support for heterogeneous storages in HDFS - DN as a collection of storages

                • Major - Major loss of function.
                • Closed

                Improvement - An improvement or enhancement to an existing feature or task. HDFS-8895 Remove deprecated BlockStorageLocation APIs

                • Major - Major loss of function.
                • Resolved

                Brainstorming - A place to record back and forth on notions not yet formed enough to make a 'New Feature' or 'Task' HBASE-6572 Tiered HFile storage

                • Major - Major loss of function.
                • Open
                is superceded by

                Improvement - An improvement or enhancement to an existing feature or task. HDFS-8887 Expose storage type and storage ID in BlockLocation

                • Major - Major loss of function.
                • Resolved

                  Activity

                    People

                    • Assignee:
                      andrew.wang Andrew Wang
                      Reporter:
                      andrew.wang Andrew Wang
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      32 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: