[HBASE-28399] region size can be wrong from RegionSizeCalculator

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-beta-1
• Fix Version/s: 3.0.0-beta-2
• Component/s: mapreduce
• Labels: pull-request-available
• Assignee: ruanhui
• Reporter: ruanhui
• Created: 23/Feb/24 09:17
• Updated: 28/Feb/24 16:58

Description

The RegionSizeCalculator calculates region byte size using the following method

private static final long MEGABYTE = 1024L * 1024L;

long regionSizeBytes =
  ((long) regionLoad.getStoreFileSize().get(Size.Unit.MEGABYTE)) * MEGABYTE; 

However, this method will lose accuracy. For example, the result of 

((long) new Size(1, Size.Unit.BYTE).get(Size.Unit.MEGABYTE)) * MEGABYTE 

is 0. This will result in a TableInputSplit with a length of 0, but in fact this TableInputSplit has a small amount of data.

 

This TableInputSplit will be ignored if we enable spark.hadoopRDD.ignoreEmptySplits.

Issue Links

• is related to: HBASE-28354 RegionSizeCalculator throws NPE when regions are in transition
• links to: GitHub Pull Request #5700