Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6338

Tests fail due to runtime profile for query with limit missing pieces

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.11.0, Impala 2.12.0
    • Fix Version/s: None
    • Component/s: Backend
    • Epic Color:
      ghx-label-7

      Description

      Stack trace:

      query_test/test_observability.py:123: in test_profile_fragment_instances
          assert results.runtime_profile.count("HDFS_SCAN_NODE") == 12
      E   assert 11 == 12
      E    +  where 11 = <built-in method count of str object at 0x68bd2f0>('HDFS_SCAN_NODE')
      E    +    where <built-in method count of str object at 0x68bd2f0> = 'Query (id=ae4cee91aafc5c6c:11b545c600000000):\n  DEBUG MODE WARNING: Query profile created while running a DEBUG buil...ontextSwitches: 0 (0)\n           - TotalRawHdfsReadTime(*): 5s784ms\n           - TotalReadThroughput: 17.33 MB/sec\n'.count
      E    +      where 'Query (id=ae4cee91aafc5c6c:11b545c600000000):\n  DEBUG MODE WARNING: Query profile created while running a DEBUG buil...ontextSwitches: 0 (0)\n           - TotalRawHdfsReadTime(*): 5s784ms\n           - TotalReadThroughput: 17.33 MB/sec\n' = <tests.beeswax.impala_beeswax.ImpalaBeeswaxResult object at 0x6322e10>.runtime_profile
      

      Query:

      with l as (select * from tpch.lineitem UNION ALL select * from tpch.lineitem)
              select STRAIGHT_JOIN count(*) from (select * from tpch.lineitem a LIMIT 1) a
              join (select * from l LIMIT 2000000) b on a.l_orderkey = -b.l_orderkey;
      

      Summary:

      Operator             #Hosts  Avg Time  Max Time  #Rows  Est. #Rows   Peak Mem  Est. Peak Mem  Detail                
      --------------------------------------------------------------------------------------------------------------------
      05:AGGREGATE              1   0.000ns   0.000ns      1           1   28.00 KB       10.00 MB  FINALIZE              
      04:HASH JOIN              1  15.000ms  15.000ms      0           1  141.06 MB       17.00 MB  INNER JOIN, BROADCAST 
      |--08:EXCHANGE            1   4s153ms   4s153ms  2.00M       2.00M          0              0  UNPARTITIONED         
      |  07:EXCHANGE            1   3s783ms   3s783ms  2.00M       2.00M          0              0  UNPARTITIONED         
      |  01:UNION               3  17.000ms  28.001ms  3.03M       2.00M          0              0                        
      |  |--03:SCAN HDFS        3   0.000ns   0.000ns      0       6.00M          0      176.00 MB  tpch.lineitem         
      |  02:SCAN HDFS           3   6s133ms   6s948ms  3.03M       6.00M   24.02 MB      176.00 MB  tpch.lineitem         
      06:EXCHANGE               1   5s655ms   5s655ms      1           1          0              0  UNPARTITIONED         
      00:SCAN HDFS              3   4s077ms   6s207ms      2           1   16.05 MB      176.00 MB  tpch.lineitem a       
      

      Plan:

      ----------------
      Max Per-Host Resource Reservation: Memory=17.00MB
      Per-Host Resource Estimates: Memory=379.00MB
      
      F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
      |  Per-Host Resources: mem-estimate=27.00MB mem-reservation=17.00MB
      PLAN-ROOT SINK
      |  mem-estimate=0B mem-reservation=0B
      |
      05:AGGREGATE [FINALIZE]
      |  output: count(*)
      |  mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
      |  tuple-ids=7 row-size=8B cardinality=1
      |
      04:HASH JOIN [INNER JOIN, BROADCAST]
      |  hash predicates: a.l_orderkey = -1 * l_orderkey
      |  fk/pk conjuncts: assumed fk/pk
      |  runtime filters: RF000[bloom] <- -1 * l_orderkey
      |  mem-estimate=17.00MB mem-reservation=17.00MB spill-buffer=1.00MB
      |  tuple-ids=0,4 row-size=16B cardinality=1
      |
      |--08:EXCHANGE [UNPARTITIONED]
      |  |  mem-estimate=0B mem-reservation=0B
      |  |  tuple-ids=4 row-size=8B cardinality=2000000
      |  |
      |  F05:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
      |  Per-Host Resources: mem-estimate=0B mem-reservation=0B
      |  07:EXCHANGE [UNPARTITIONED]
      |  |  limit: 2000000
      |  |  mem-estimate=0B mem-reservation=0B
      |  |  tuple-ids=4 row-size=8B cardinality=2000000
      |  |
      |  F04:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
      |  Per-Host Resources: mem-estimate=176.00MB mem-reservation=0B
      |  01:UNION
      |  |  pass-through-operands: all
      |  |  limit: 2000000
      |  |  mem-estimate=0B mem-reservation=0B
      |  |  tuple-ids=4 row-size=8B cardinality=2000000
      |  |
      |  |--03:SCAN HDFS [tpch.lineitem, RANDOM]
      |  |     partitions=1/1 files=1 size=718.94MB
      |  |     stored statistics:
      |  |       table: rows=6001215 size=718.94MB
      |  |       columns: all
      |  |     extrapolated-rows=disabled
      |  |     mem-estimate=176.00MB mem-reservation=0B
      |  |     tuple-ids=3 row-size=8B cardinality=6001215
      |  |
      |  02:SCAN HDFS [tpch.lineitem, RANDOM]
      |     partitions=1/1 files=1 size=718.94MB
      |     stored statistics:
      |       table: rows=6001215 size=718.94MB
      |       columns: all
      |     extrapolated-rows=disabled
      |     mem-estimate=176.00MB mem-reservation=0B
      |     tuple-ids=2 row-size=8B cardinality=6001215
      |
      06:EXCHANGE [UNPARTITIONED]
      |  limit: 1
      |  mem-estimate=0B mem-reservation=0B
      |  tuple-ids=0 row-size=8B cardinality=1
      |
      F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
      Per-Host Resources: mem-estimate=176.00MB mem-reservation=0B
      00:SCAN HDFS [tpch.lineitem a, RANDOM]
         partitions=1/1 files=1 size=718.94MB
         runtime filters: RF000[bloom] -> a.l_orderkey
         stored statistics:
           table: rows=6001215 size=718.94MB
           columns: all
         extrapolated-rows=disabled
         limit: 1
         mem-estimate=176.00MB mem-reservation=0B
         tuple-ids=0 row-size=8B cardinality=1
      ----------------
      

        Attachments

        1. profile-success.txt
          59 kB
          Michael Brown
        2. profile-failure.txt
          56 kB
          Michael Brown

          Issue Links

            Activity

              People

              • Assignee:
                twmarshall Thomas Tauber-Marshall
                Reporter:
                dknupp David Knupp
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: