[IMPALA-11278] Cardinality of small HBase regions is overestimated since HBASE-26340

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 4.1.0
• Component/s: Catalog, Frontend
• Epic Color: ghx-label-3
• Assignee: Unassigned
• Reporter: Csaba Ringhofer
• Created: 02/May/22 12:38
• Updated: 02/May/22 13:03

Description

Impala uses the size of an HBase region to estimate the number of rows, and the API we use (https://hbase.apache.org/2.4/apidocs/org/apache/hadoop/hbase/RegionLoad.html#getStorefileSizeMB() ) returns a size at MB precision. Since HBASE-26340 it returns 1 instead of 0 for very small but not empty tables, which leads to massively overestimating its size (we handle 0 in a special way. so we didn't estimate row count as 0: https://github.com/apache/impala/blob/78609dca32d8ce996247c9552ba676a853c74686/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java#L585 )

In newer versions of HBase getStorefileSizeMB() is deprecated and there are functions to get the size at byte granulity. Using it could solve the massive overestimation, but it may make our planner tests more sensitive to small size changes in HBase regions.

HBASE-26340 is included in the Impala's HBase dependency since https://github.com/apache/impala/commit/ca48b940ec6281d492ad525418f234308a82eedf

Issue Links

• relates to: IMPALA-8024 HBase table cardinality estimates are wrong
• relates to: HBASE-26340 TableSplit returns false size under 1MB