VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.1.0
    • None
    • None
    • Reviewed

    Description

      As opposed to local blocks, each DN keeps track of all blocks in PROVIDED storage. This can be millions of blocks for 100s of TBs of PROVIDED data. Storing the data for these blocks can lead to a large memory footprint. Further, with so many blocks, DirectoryScanner running on a PROVIDED volume can increase the memory and CPU utilization.

      To reduce these overheads, this JIRA aims to (a) disable the DirectoryScanner on PROVIDED volumes (as HDFS-9806 focuses on only read-only data in PROVIDED volumes), (b) reduce the space occupied by FinalizedProvidedReplicaInfo by using a common URI prefix across all PROVIDED blocks.

      Attachments

        1. HDFS-12777-HDFS-9806.001.patch
          15 kB
          Virajith Jalaparti
        2. HDFS-12777-HDFS-9806.002.patch
          15 kB
          Virajith Jalaparti
        3. HDFS-12777-HDFS-9806.003.patch
          18 kB
          Virajith Jalaparti
        4. HDFS-12777-HDFS-9806.004.patch
          18 kB
          Virajith Jalaparti

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            virajith Virajith Jalaparti
            virajith Virajith Jalaparti
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment