Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14531

Datanode's ScanInfo requires excessive memory

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.0.0-alpha
    • None
    • hdfs
    • None

    Description

      The DirectoryScanner's ScanInfo map consumes ~4.5X memory as replicas as the replica map.  For 1.1M replicas: the replica map is ~91M while the scan info is ~405M.

      Attachments

        Issue Links

        is related to

        Bug - A problem which impairs or prevents the functions of the product. HDFS-15621 Datanode DirectoryScanner uses excessive memory

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. HDFS-13883 Reduce memory consumption and GC of directory scan

        • Major - Major loss of function.
        • Open

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            daryn Daryn Sharp
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:

              Slack

                Issue deployment