[HBASE-25226] Optimize in-memory representation for HBase map reduce table splits for MultiTableInputFormat

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Sandeep Pal
• Reporter: Sandeep Pal
• Created: 28/Oct/20 17:45
• Updated: 28/Oct/20 17:45

Description

It has been observed that when the table has too many regions, MR jobs consume a lot of memory in the client. This is because we keep the region level information in memory and the memory heavy object is TableSplit because of the Scan object as a part of it.

There is a jira HBASE-24859 which fix this single table TableInputFormat because we do not use the scan object from TableSplit in this case.
However, it looks like we can do some optimization in case of MultiTableInputFormat as well since each split is not required to have memory heavy scan object.

Issue Links

• is a child of: HBASE-24859 Optimize in-memory representation of mapreduce TableSplit objects