Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9260

Improve the performance and GC friendliness of NameNode startup and full block reports

    XMLWordPrintableJSON

Details

    Description

      This patch changes the datastructures used for BlockInfos and Replicas to keep them sorted. This allows faster and more GC friendly handling of full block reports.

      Would like to hear peoples feedback on this change.

      Attachments

        1. FBR processing.png
          179 kB
          Daryn Sharp
        2. HDFS-7435.001.patch
          105 kB
          Staffan Friberg
        3. HDFS-7435.002.patch
          104 kB
          Staffan Friberg
        4. HDFS-7435.003.patch
          104 kB
          Staffan Friberg
        5. HDFS-7435.004.patch
          105 kB
          Staffan Friberg
        6. HDFS-7435.005.patch
          144 kB
          Staffan Friberg
        7. HDFS-7435.006.patch
          144 kB
          Staffan Friberg
        8. HDFS-7435.007.patch
          145 kB
          Staffan Friberg
        9. HDFS-9260.008.patch
          145 kB
          Staffan Friberg
        10. HDFS-9260.009.patch
          145 kB
          Staffan Friberg
        11. HDFS-9260.010.patch
          153 kB
          Staffan Friberg
        12. HDFS-9260.011.patch
          154 kB
          Staffan Friberg
        13. HDFS-9260.012.patch
          150 kB
          Staffan Friberg
        14. HDFS-9260.013.patch
          150 kB
          Staffan Friberg
        15. HDFS-9260.014.patch
          150 kB
          Staffan Friberg
        16. HDFS-9260.015.patch
          150 kB
          Staffan Friberg
        17. HDFS-9260.016.patch
          146 kB
          Staffan Friberg
        18. HDFS-9260.017.patch
          148 kB
          Staffan Friberg
        19. HDFS-9260.018.patch
          147 kB
          Staffan Friberg
        20. HDFSBenchmarks.zip
          25 kB
          Staffan Friberg
        21. HDFSBenchmarks2.zip
          32 kB
          Staffan Friberg
        22. HDFS Block and Replica Management 20151013.pdf
          143 kB
          Staffan Friberg

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HDFS-12363 Possible NPE in BlockManager$StorageInfoDefragmenter#scanAndCompactStorages

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-13671 Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HDFS-7967 Reduce the performance impact of the balancer

          • Critical - Crashes, loss of data, severe memory leak.
          • Patch Available

          Bug - A problem which impairs or prevents the functions of the product. HDFS-14680 StorageInfoDefragmenter should handle exceptions gently

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HDFS-15131 FoldedTreeSet appears to degrade over time

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-11310 Reduce the performance impact of the balancer (trunk port)

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Activity

            People

              sfriberg Staffan Friberg
              sfriberg Staffan Friberg
              Votes:
              0 Vote for this issue
              Watchers:
              44 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: