[HDFS-12777] [READ] Reduce memory and CPU footprint for PROVIDED volumes.

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1.0
• Hadoop Flags: Reviewed
• Assignee: Virajith Jalaparti
• Reporter: Virajith Jalaparti
• Created: 03/Nov/17 23:56
• Updated: 17/Dec/20 14:09
• Resolved: 10/Nov/17 18:22

Description

As opposed to local blocks, each DN keeps track of all blocks in PROVIDED storage. This can be millions of blocks for 100s of TBs of PROVIDED data. Storing the data for these blocks can lead to a large memory footprint. Further, with so many blocks, DirectoryScanner running on a PROVIDED volume can increase the memory and CPU utilization.

To reduce these overheads, this JIRA aims to (a) disable the DirectoryScanner on PROVIDED volumes (as HDFS-9806 focuses on only read-only data in PROVIDED volumes), (b) reduce the space occupied by FinalizedProvidedReplicaInfo by using a common URI prefix across all PROVIDED blocks.

Issue Links

• is related to: HDFS-15734 [READ] DirectoryScanner#scan need not check StorageType.PROVIDED