[HBASE-24859] Optimize in-memory representation of mapreduce TableSplit objects

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 2.2.7
• Fix Version/s: 3.0.0-alpha-1, 1.7.0, 2.4.0, 2.2.7, 2.3.4
• Component/s: mapreduce
• Assignee: Sandeep Pal
• Reporter: Sandeep Pal
• Created: 11/Aug/20 18:41
• Updated: 04/Nov/20 18:14
• Resolved: 03/Nov/20 16:05

Description

It has been observed that when the table has too many regions, MR jobs consume a lot of memory in the client. This is because we keep the region level information in memory and the memory heavy object is TableSplit because of the Scan object as a part of it.

However, it looks like the TableInputFormat for single table doesn't need to store the scan object in the TableSplit because we do not use it and all the splits are expected to have the exact same scan object. In TableInputFormat we use the scan object directly from the MR conf.

Issue Links

• is a parent of: HBASE-25226 Optimize in-memory representation for HBase map reduce table splits for MultiTableInputFormat
• links to: GitHub Pull Request #2591
• links to: GitHub Pull Request #2609
• links to: GitHub Pull Request #2610