Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9260

Improve the performance and GC friendliness of NameNode startup and full block reports

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      This patch changes the datastructures used for BlockInfos and Replicas to keep them sorted. This allows faster and more GC friendly handling of full block reports.

      Would like to hear peoples feedback on this change.

      Attachments

        1. FBR processing.png
          179 kB
          Daryn Sharp
        2. HDFS-7435.001.patch
          105 kB
          Staffan Friberg
        3. HDFS-7435.002.patch
          104 kB
          Staffan Friberg
        4. HDFS-7435.003.patch
          104 kB
          Staffan Friberg
        5. HDFS-7435.004.patch
          105 kB
          Staffan Friberg
        6. HDFS-7435.005.patch
          144 kB
          Staffan Friberg
        7. HDFS-7435.006.patch
          144 kB
          Staffan Friberg
        8. HDFS-7435.007.patch
          145 kB
          Staffan Friberg
        9. HDFS-9260.008.patch
          145 kB
          Staffan Friberg
        10. HDFS-9260.009.patch
          145 kB
          Staffan Friberg
        11. HDFS-9260.010.patch
          153 kB
          Staffan Friberg
        12. HDFS-9260.011.patch
          154 kB
          Staffan Friberg
        13. HDFS-9260.012.patch
          150 kB
          Staffan Friberg
        14. HDFS-9260.013.patch
          150 kB
          Staffan Friberg
        15. HDFS-9260.014.patch
          150 kB
          Staffan Friberg
        16. HDFS-9260.015.patch
          150 kB
          Staffan Friberg
        17. HDFS-9260.016.patch
          146 kB
          Staffan Friberg
        18. HDFS-9260.017.patch
          148 kB
          Staffan Friberg
        19. HDFS-9260.018.patch
          147 kB
          Staffan Friberg
        20. HDFSBenchmarks.zip
          25 kB
          Staffan Friberg
        21. HDFSBenchmarks2.zip
          32 kB
          Staffan Friberg
        22. HDFS Block and Replica Management 20151013.pdf
          143 kB
          Staffan Friberg

        Issue Links

        breaks

        Bug - A problem which impairs or prevents the functions of the product. HDFS-12363 Possible NPE in BlockManager$StorageInfoDefragmenter#scanAndCompactStorages

        • Major - Major loss of function.
        • Resolved
        is related to

        Bug - A problem which impairs or prevents the functions of the product. HDFS-13671 Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet

        • Major - Major loss of function.
        • Resolved
        relates to

        Sub-task - The sub-task of the issue HDFS-7967 Reduce the performance impact of the balancer

        • Critical - Crashes, loss of data, severe memory leak.
        • Patch Available

        Bug - A problem which impairs or prevents the functions of the product. HDFS-14680 StorageInfoDefragmenter should handle exceptions gently

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. HDFS-15131 FoldedTreeSet appears to degrade over time

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. HDFS-11310 Reduce the performance impact of the balancer (trunk port)

        • Critical - Crashes, loss of data, severe memory leak.
        • Open

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sfriberg Staffan Friberg Assign to me
            sfriberg Staffan Friberg
            Votes:
            0 Vote for this issue
            Watchers:
            44 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment