[IMPALA-11132] Front-end test PlannerTest.testResourceRequirements can fail

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Target Version: Impala 4.0.0
• Epic Color: ghx-label-5
• Assignee: Qifan Chen
• Reporter: Qifan Chen
• Created: 17/Feb/22 22:15
• Updated: 24/Feb/22 17:11
• Resolved: 24/Feb/22 17:11

Description

The test miscalculates per-host memory requirements, apparently due to an incorrect HBase cardinality estimate:

Section DISTRIBUTEDPLAN of query:
select * from functional_hbase.alltypessmall

Actual does not match expected result:
Max Per-Host Resource Reservation: Memory=4.00MB Threads=2
Per-Host Resource Estimates: Memory=10MB
Codegen disabled by planner
Analyzed query: SELECT * FROM functional_hbase.alltypessmall

F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
|  Per-Host Resources: mem-estimate=5.08MB mem-reservation=4.00MB thread-reservation=1
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PLAN-ROOT SINK
|  output exprs: functional_hbase.alltypessmall.id, functional_hbase.alltypessmall.bigint_col, functional_hbase.alltypessmall.bool_col, functional_hbase.alltypessmall.date_string_col, functional_hbase.alltypessmall.double_col, functional_hbase.alltypessmall.float_col, functional_hbase.alltypessmall.int_col, functional_hbase.alltypessmall.month, functional_hbase.alltypessmall.smallint_col, functional_hbase.alltypessmall.string_col, functional_hbase.alltypessmall.timestamp_col, functional_hbase.alltypessmall.tinyint_col, functional_hbase.alltypessmall.year
|  mem-estimate=4.00MB mem-reservation=4.00MB spill-buffer=2.00MB thread-reservation=0
|
01:EXCHANGE [UNPARTITIONED]
|  mem-estimate=1.08MB mem-reservation=0B thread-reservation=0
|  tuple-ids=0 row-size=89B cardinality=28.57K
|  in pipelines: 00(GETNEXT)
|
F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
Per-Host Resources: mem-estimate=4.00KB mem-reservation=0B thread-reservation=1
00:SCAN HBASE [functional_hbase.alltypessmall]
   stored statistics:
     table: rows=100
     columns: all
   mem-estimate=4.00KB mem-reservation=0B thread-reservation=0
   tuple-ids=0 row-size=89B cardinality=28.57K
   in pipelines: 00(GETNEXT)

Expected:
Max Per-Host Resource Reservation: Memory=4.00MB Threads=2
Per-Host Resource Estimates: Memory=10MB
Codegen disabled by planner
Analyzed query: SELECT * FROM functional_hbase.alltypessmall

F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
|  Per-Host Resources: mem-estimate=4.02MB mem-reservation=4.00MB thread-reservation=1
PLAN-ROOT SINK
|  output exprs: functional_hbase.alltypessmall.id, functional_hbase.alltypessmall.bigint_col, functional_hbase.alltypessmall.bool_col, functional_hbase.alltypessmall.date_string_col, functional_hbase.alltypessmall.double_col, functional_hbase.alltypessmall.float_col, functional_hbase.alltypessmall.int_col, functional_hbase.alltypessmall.month, functional_hbase.alltypessmall.smallint_col, functional_hbase.alltypessmall.string_col, functional_hbase.alltypessmall.timestamp_col, functional_hbase.alltypessmall.tinyint_col, functional_hbase.alltypessmall.year
|  mem-estimate=4.00MB mem-reservation=4.00MB spill-buffer=2.00MB thread-reservation=0
|
01:EXCHANGE [UNPARTITIONED]
|  mem-estimate=16.00KB mem-reservation=0B thread-reservation=0
|  tuple-ids=0 row-size=89B cardinality=50
|  in pipelines: 00(GETNEXT)
|
F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
Per-Host Resources: mem-estimate=4.00KB mem-reservation=0B thread-reservation=1
00:SCAN HBASE [functional_hbase.alltypessmall]
   stored statistics:
     table: rows=100
     columns: all
   mem-estimate=4.00KB mem-reservation=0B thread-reservation=0
   tuple-ids=0 row-size=89B cardinality=50
   in pipelines: 00(GETNEXT)