XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As opposed to local blocks, each DN keeps track of all blocks in PROVIDED storage. This can be millions of blocks for 100s of TBs of PROVIDED data. Storing the data for these blocks can lead to a large memory footprint. Further, with so many blocks, DirectoryScanner running on a PROVIDED volume can increase the memory and CPU utilization.

      To reduce these overheads, this JIRA aims to (a) disable the DirectoryScanner on PROVIDED volumes (as HDFS-9806 focuses on only read-only data in PROVIDED volumes), (b) reduce the space occupied by FinalizedProvidedReplicaInfo by using a common URI prefix across all PROVIDED blocks.

        Attachments

        1. HDFS-12777-HDFS-9806.004.patch
          18 kB
          Virajith Jalaparti
        2. HDFS-12777-HDFS-9806.003.patch
          18 kB
          Virajith Jalaparti
        3. HDFS-12777-HDFS-9806.002.patch
          15 kB
          Virajith Jalaparti
        4. HDFS-12777-HDFS-9806.001.patch
          15 kB
          Virajith Jalaparti

          Activity

            People

            • Assignee:
              virajith Virajith Jalaparti
              Reporter:
              virajith Virajith Jalaparti
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: