Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4638

Queries with mt_dop > 0 skip admission control

    Details

      Description

      With mt_dop > 0 queries skip admission control

      Query (id=c34b4445491d7984:4514bfc000000000):
        Summary:
          Session ID: 7743a91e02573fae:a8f116736e980bad
          Session Type: BEESWAX
          Start Time: 2016-12-09 14:26:54.031855000
          End Time: 2016-12-09 14:27:00.713773000
          Query Type: QUERY
          Query State: FINISHED
          Query Status: OK
          Impala Version: impalad version 2.8.0-SNAPSHOT RELEASE (build a5176818172103693fa124d69982963b04b1098e)
          User: systest@HALXG.CLOUDERA.COM
          Connected User: systest@HALXG.CLOUDERA.COM
          Delegated User: 
          Network Address: 10.17.234.19:60064
          Default Db: tpch_300_parquet
          Sql Statement: select count(*) from lineitem group by l_orderkey having count(*) > 10000
          Coordinator: vd1309.halxg.cloudera.com:22000
          Query Options (non default): NUM_SCANNER_THREADS=1,MT_DOP=8
          Plan: 
      ----------------
      Estimated Per-Host Requirements: Memory=7.98GB VCores=2
      WARNING: Spilling is disabled for this query as a safety guard.
      Reason: Query option disable_unsafe_spills is set, at least one table
      is missing relevant stats, and no plan hints were given.
      
      PLAN-ROOT SINK
      |
      04:EXCHANGE [UNPARTITIONED]
      |  hosts=7 per-host-mem=unavailable
      |  tuple-ids=1 row-size=16B cardinality=43790950
      |
      03:AGGREGATE [FINALIZE]
      |  output: count:merge(*)
      |  group by: l_orderkey
      |  having: count(*) > 10000
      |  hosts=7 per-host-mem=735.02MB
      |  tuple-ids=1 row-size=16B cardinality=43790950
      |
      02:EXCHANGE [HASH(l_orderkey)]
      |  hosts=7 per-host-mem=0B
      |  tuple-ids=1 row-size=16B cardinality=437909504
      |
      01:AGGREGATE [STREAMING]
      |  output: count(*)
      |  group by: l_orderkey
      |  hosts=7 per-host-mem=7.18GB
      |  tuple-ids=1 row-size=16B cardinality=437909504
      |
      00:SCAN HDFS [tpch_300_parquet.lineitem, RANDOM]
         partitions=1/1 files=259 size=63.71GB
         table stats: 1799989091 rows total
         column stats: all
         hosts=7 per-host-mem=88.00MB
         tuple-ids=0 row-size=8B cardinality=1799989091
      ----------------
          Estimated Per-Host Mem: 8570202713
          Estimated Per-Host VCores: 2
          Request Pool: root.systest
          ExecSummary: 
      Operator       #Hosts   Avg Time   Max Time    #Rows  Est. #Rows   Peak Mem  Est. Peak Mem  Detail                    
      ----------------------------------------------------------------------------------------------------------------------
      04:EXCHANGE         1    7.533us    7.533us        0      43.79M          0        -1.00 B  UNPARTITIONED             
      03:AGGREGATE       56    2s643ms    2s861ms        0      43.79M  402.06 MB      735.02 MB  FINALIZE                  
      02:EXCHANGE        56  276.883ms  301.636ms  450.04M     437.91M          0              0  HASH(l_orderkey)          
      01:AGGREGATE       56    2s189ms    2s519ms  450.04M     437.91M  521.33 MB        7.18 GB  STREAMING                 
      00:SCAN HDFS       56  506.329ms  649.212ms    1.80B       1.80B   12.47 MB       88.00 MB  tpch_300_parquet.lineitem 
          Planner Timeline: 2.319ms
             - Analysis finished: 674.915us (674.915us)
             - Equivalence classes computed: 766.195us (91.280us)
             - Single node plan created: 1.193ms (427.359us)
             - Runtime filters computed: 1.222ms (29.215us)
             - Distributed plan created: 1.372ms (149.684us)
             - Lineage info computed: 1.413ms (41.209us)
             - Parallel plans created: 1.465ms (51.646us)
             - Planning finished: 2.319ms (854.253us)
          Query Timeline: 6s705ms
             - Query submitted: 70.897us (70.897us)
             - Planning finished: 3.788ms (3.717ms)
             - Ready to start 113 fragment instances: 5.457ms (1.669ms)
             - All 113 fragment instances started: 157.998ms (152.540ms)
             - Rows available: 6s539ms (6s381ms)
             - First row fetched: 6s627ms (87.972ms)
             - Unregister query: 6s681ms (54.351ms)
           - ComputeScanRangeAssignmentTimer: 445.101us
      

        Activity

        Hide
        alex.behm Alexander Behm added a comment -

        commit 48792eb92f010af99246a14d5a3570dfed662dfd
        Author: Alex Behm <alex.behm@cloudera.com>
        Date: Fri Dec 9 14:51:17 2016 -0800

        IMPALA-4638: Run queries with MT_DOP through admission control.

        The bug was a simple oversight: A TODO that was not addressed.

        Testing: I examined query profiles before and after this
        change with and without MT_DOP set. After this patch all
        queries (with and without MT_DOP) go through admission control.

        Change-Id: I3e05697a59e14617fb6a13ce8e0410820823fd3a
        Reviewed-on: http://gerrit.cloudera.org:8080/5447
        Reviewed-by: Alex Behm <alex.behm@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        alex.behm Alexander Behm added a comment - commit 48792eb92f010af99246a14d5a3570dfed662dfd Author: Alex Behm <alex.behm@cloudera.com> Date: Fri Dec 9 14:51:17 2016 -0800 IMPALA-4638 : Run queries with MT_DOP through admission control. The bug was a simple oversight: A TODO that was not addressed. Testing: I examined query profiles before and after this change with and without MT_DOP set. After this patch all queries (with and without MT_DOP) go through admission control. Change-Id: I3e05697a59e14617fb6a13ce8e0410820823fd3a Reviewed-on: http://gerrit.cloudera.org:8080/5447 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Internal Jenkins
        Hide
        jbapple Jim Apple added a comment -

        This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09.

        2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that:

        https://github.com/apache/incubator-impala/commits/2.8.0

        Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0?

        Thank you!

        Show
        jbapple Jim Apple added a comment - This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09. 2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that: https://github.com/apache/incubator-impala/commits/2.8.0 Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0? Thank you!

          People

          • Assignee:
            alex.behm Alexander Behm
            Reporter:
            mmokhtar Mostafa Mokhtar
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development