[IMPALA-6338] Tests fail due to runtime profile for query with limit missing pieces

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.11.0, Impala 2.12.0
• Component/s: Backend
• Labels: broken-build flaky
• Epic Color: ghx-label-7
• Assignee: Thomas Tauber-Marshall
• Reporter: David Knupp
• Created: 18/Dec/17 20:17
• Updated: 07/Jun/18 18:59
• Resolved: 07/Jun/18 18:59

Description

Stack trace:

query_test/test_observability.py:123: in test_profile_fragment_instances
    assert results.runtime_profile.count("HDFS_SCAN_NODE") == 12
E   assert 11 == 12
E    +  where 11 = <built-in method count of str object at 0x68bd2f0>('HDFS_SCAN_NODE')
E    +    where <built-in method count of str object at 0x68bd2f0> = 'Query (id=ae4cee91aafc5c6c:11b545c600000000):\n  DEBUG MODE WARNING: Query profile created while running a DEBUG buil...ontextSwitches: 0 (0)\n           - TotalRawHdfsReadTime(*): 5s784ms\n           - TotalReadThroughput: 17.33 MB/sec\n'.count
E    +      where 'Query (id=ae4cee91aafc5c6c:11b545c600000000):\n  DEBUG MODE WARNING: Query profile created while running a DEBUG buil...ontextSwitches: 0 (0)\n           - TotalRawHdfsReadTime(*): 5s784ms\n           - TotalReadThroughput: 17.33 MB/sec\n' = <tests.beeswax.impala_beeswax.ImpalaBeeswaxResult object at 0x6322e10>.runtime_profile

Query:

with l as (select * from tpch.lineitem UNION ALL select * from tpch.lineitem)
        select STRAIGHT_JOIN count(*) from (select * from tpch.lineitem a LIMIT 1) a
        join (select * from l LIMIT 2000000) b on a.l_orderkey = -b.l_orderkey;

Summary:

Operator             #Hosts  Avg Time  Max Time  #Rows  Est. #Rows   Peak Mem  Est. Peak Mem  Detail                
--------------------------------------------------------------------------------------------------------------------
05:AGGREGATE              1   0.000ns   0.000ns      1           1   28.00 KB       10.00 MB  FINALIZE              
04:HASH JOIN              1  15.000ms  15.000ms      0           1  141.06 MB       17.00 MB  INNER JOIN, BROADCAST 
|--08:EXCHANGE            1   4s153ms   4s153ms  2.00M       2.00M          0              0  UNPARTITIONED         
|  07:EXCHANGE            1   3s783ms   3s783ms  2.00M       2.00M          0              0  UNPARTITIONED         
|  01:UNION               3  17.000ms  28.001ms  3.03M       2.00M          0              0                        
|  |--03:SCAN HDFS        3   0.000ns   0.000ns      0       6.00M          0      176.00 MB  tpch.lineitem         
|  02:SCAN HDFS           3   6s133ms   6s948ms  3.03M       6.00M   24.02 MB      176.00 MB  tpch.lineitem         
06:EXCHANGE               1   5s655ms   5s655ms      1           1          0              0  UNPARTITIONED         
00:SCAN HDFS              3   4s077ms   6s207ms      2           1   16.05 MB      176.00 MB  tpch.lineitem a       

Plan:

----------------
Max Per-Host Resource Reservation: Memory=17.00MB
Per-Host Resource Estimates: Memory=379.00MB

F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
|  Per-Host Resources: mem-estimate=27.00MB mem-reservation=17.00MB
PLAN-ROOT SINK
|  mem-estimate=0B mem-reservation=0B
|
05:AGGREGATE [FINALIZE]
|  output: count(*)
|  mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
|  tuple-ids=7 row-size=8B cardinality=1
|
04:HASH JOIN [INNER JOIN, BROADCAST]
|  hash predicates: a.l_orderkey = -1 * l_orderkey
|  fk/pk conjuncts: assumed fk/pk
|  runtime filters: RF000[bloom] <- -1 * l_orderkey
|  mem-estimate=17.00MB mem-reservation=17.00MB spill-buffer=1.00MB
|  tuple-ids=0,4 row-size=16B cardinality=1
|
|--08:EXCHANGE [UNPARTITIONED]
|  |  mem-estimate=0B mem-reservation=0B
|  |  tuple-ids=4 row-size=8B cardinality=2000000
|  |
|  F05:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
|  Per-Host Resources: mem-estimate=0B mem-reservation=0B
|  07:EXCHANGE [UNPARTITIONED]
|  |  limit: 2000000
|  |  mem-estimate=0B mem-reservation=0B
|  |  tuple-ids=4 row-size=8B cardinality=2000000
|  |
|  F04:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
|  Per-Host Resources: mem-estimate=176.00MB mem-reservation=0B
|  01:UNION
|  |  pass-through-operands: all
|  |  limit: 2000000
|  |  mem-estimate=0B mem-reservation=0B
|  |  tuple-ids=4 row-size=8B cardinality=2000000
|  |
|  |--03:SCAN HDFS [tpch.lineitem, RANDOM]
|  |     partitions=1/1 files=1 size=718.94MB
|  |     stored statistics:
|  |       table: rows=6001215 size=718.94MB
|  |       columns: all
|  |     extrapolated-rows=disabled
|  |     mem-estimate=176.00MB mem-reservation=0B
|  |     tuple-ids=3 row-size=8B cardinality=6001215
|  |
|  02:SCAN HDFS [tpch.lineitem, RANDOM]
|     partitions=1/1 files=1 size=718.94MB
|     stored statistics:
|       table: rows=6001215 size=718.94MB
|       columns: all
|     extrapolated-rows=disabled
|     mem-estimate=176.00MB mem-reservation=0B
|     tuple-ids=2 row-size=8B cardinality=6001215
|
06:EXCHANGE [UNPARTITIONED]
|  limit: 1
|  mem-estimate=0B mem-reservation=0B
|  tuple-ids=0 row-size=8B cardinality=1
|
F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
Per-Host Resources: mem-estimate=176.00MB mem-reservation=0B
00:SCAN HDFS [tpch.lineitem a, RANDOM]
   partitions=1/1 files=1 size=718.94MB
   runtime filters: RF000[bloom] -> a.l_orderkey
   stored statistics:
     table: rows=6001215 size=718.94MB
     columns: all
   extrapolated-rows=disabled
   limit: 1
   mem-estimate=176.00MB mem-reservation=0B
   tuple-ids=0 row-size=8B cardinality=1
----------------

Issue Links

• breaks: IMPALA-6484 Crash in impala::RuntimeProfile::SortChildren
• is blocked by: IMPALA-2990 Coordinator should timeout and cancel queries with unresponsive / stuck executors
• is blocked by: IMPALA-5384 Simplify coordinator locking protocol
• is duplicated by: IMPALA-6535 test_profile_fragment_instances failure: incomplete query profile
• is duplicated by: IMPALA-6960 TestMtDopParquet.test_parquet_filtering flaky
• is related to: IMPALA-341 Remote profiles may be ignored by coordinator if query has a limit
• is related to: IMPALA-6960 TestMtDopParquet.test_parquet_filtering flaky