[HBASE-3693] isMajorCompaction() check triggers lots of listStatus DFS RPC calls from HBase

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Hadoop Flags: Reviewed
• Assignee: Liyin Tang
• Reporter: Kannan Muthukkaruppan
• Created: 23/Mar/11 17:37
• Updated: 12/Jun/22 00:55
• Resolved: 25/Mar/11 21:04

Description

We noticed that there are lots of listStatus calls on the ColumnFamily directories within each region, coming from this codepath:

compactionSelection()
 --> isMajorCompaction 
    --> getLowestTimestamp()
       -->  FileStatus[] stats = fs.listStatus(p);

So on every compactionSelection() we're taking this hit. While not immediately an issue, just from log inspection, this accounts for quite a large number of RPCs to namenode at the moment and seems like an unnecessary load to be sending to the namenode.

Seems like it would be easy to cache the timestamp for each opened/created StoreFile, in memory, in the region server, and avoid going to DFS each time for this information.