Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5229

Try using TCMalloc + Huge Pages for buffers

    Details

      Description

      As well as mmap() + huge pages, we could support TCMalloc + huge pages. I believe TCMalloc will support this well because:

      • We can allocate huge-page-aligned memory via posix_memalign()
      • TCMalloc, by default, always decommits large allocations upon freeing them. So if we undo the HUGEPAGE madvise() before handing the pages back to TCMalloc, then TCMalloc will decommit the huge pages backing the allocation, and we won't get TCMalloc's page heap into a weird state.

      This could give us the perf benefits of huge pages without some of the headaches associated with using mmap() directly.

        Issue Links

          Activity

          Hide
          tarmstrong Tim Armstrong added a comment -

          IMPALA-5229: huge page-backed buffers with TCMalloc

          This commit enables transparent huge pages when we're
          allocating via malloc(), not just mmap(). This gives
          us the perf benefits of huge pages, without the
          challenge that the mmap() path presented - the overhead
          of mapping and unmapping memory and the difficulty in
          reasoning about peak virtual memory consumption.

          Also sneak in some cleanup - use Rvalue refs for
          BufferHandle methods where appropriate.

          Testing:
          Updated backend tests to ensure this combination is covered.
          Ran some end-to-end tests and stress tests on my buffer pool
          dev branch and all looks good.

          Perf:
          Compared to current master, this provides a pretty clear perf
          benefit: I ran benchmarks on a single daemon with a reasonably
          large TPC-H scale factor. Large aggregations are much faster
          and everything else is the same (within variance) or slightly
          faster.

          Report Generated on 2017-04-18
          Run Description: "Base: 68f32e52bc42bef578330a4fe0edc5b292891eea vs Ref: f39d69bcd8bdc7d6d4fb42ef19966a26dea3a29d"

          Cluster Name: UNKNOWN
          Lab Run Info: UNKNOWN
          Impala Version: impalad version 2.9.0-SNAPSHOT RELEASE ()
          Baseline Impala Version: impalad version 2.9.0-SNAPSHOT RELEASE (2017-04-06)

          ----------------------------------------------------------------------------------+

          Workload File Format Avg (s) Delta(Avg) GeoMean(s) Delta(GeoMean)

          ----------------------------------------------------------------------------------+

          TARGETED-PERF(_60) parquet / none / none 19.30 -3.05% 4.91 -0.91%

          ----------------------------------------------------------------------------------+

          ------------------------------------------------------------------------------------------------------------------------------------------------------------------+

          Workload Query File Format Avg(s) Base Avg(s) Delta(Avg) StdDev(%) Base StdDev(%) Num Clients Iters

          ------------------------------------------------------------------------------------------------------------------------------------------------------------------+

          TARGETED-PERF(_60) PERF_LIMIT-Q1 parquet / none / none 0.01 0.01 R +22.95% 6.12% 2.30% 1 5
          TARGETED-PERF(_60) primitive_topn_bigint parquet / none / none 5.14 4.66 +10.35% 5.00%
          • 13.39% *
          1 5
          TARGETED-PERF(_60) primitive_conjunct_ordering_4 parquet / none / none 0.24 0.23 +8.12%
          • 12.81% *
          1.76% 1 5
          TARGETED-PERF(_60) primitive_broadcast_join_3 parquet / none / none 7.86 7.39 +6.44% 1.49% 1.41% 1 5
          TARGETED-PERF(_60) PERF_STRING-Q6 parquet / none / none 10.53 10.30 +2.24% 0.61% 0.30% 1 5
          TARGETED-PERF(_60) primitive_conjunct_ordering_5 parquet / none / none 17.23 16.90 +1.90% 1.61% 1.05% 1 5
          TARGETED-PERF(_60) primitive_conjunct_ordering_3 parquet / none / none 3.19 3.13 +1.81% 1.47% 0.45% 1 5
          TARGETED-PERF(_60) primitive_filter_bigint_non_selective parquet / none / none 1.08 1.06 +1.60% 0.26% 2.07% 1 5
          TARGETED-PERF(_60) PERF_STRING-Q3 parquet / none / none 3.75 3.71 +1.14% 0.39% 0.80% 1 5
          TARGETED-PERF(_60) primitive_broadcast_join_2 parquet / none / none 5.15 5.09 +1.11% 1.18% 0.89% 1 5
          TARGETED-PERF(_60) PERF_STRING-Q2 parquet / none / none 3.47 3.44 +1.03% 1.27% 0.61% 1 5
          TARGETED-PERF(_60) PERF_AGG-Q1 parquet / none / none 2.53 2.51 +1.01% 1.75% 1.91% 1 5
          TARGETED-PERF(_60) PERF_STRING-Q7 parquet / none / none 8.37 8.31 +0.81% 0.49% 0.58% 1 5
          TARGETED-PERF(_60) primitive_filter_string_non_selective parquet / none / none 1.90 1.88 +0.74% 1.81% 0.73% 1 5
          TARGETED-PERF(_60) primitive_filter_string_like parquet / none / none 14.82 14.73 +0.62% 0.17% 0.02% 1 5
          TARGETED-PERF(_60) primitive_top-n_all parquet / none / none 40.93 40.69 +0.61% 0.09% 0.15% 1 5
          TARGETED-PERF(_60) PERF_STRING-Q4 parquet / none / none 3.76 3.74 +0.60% 0.79% 0.53% 1 5
          TARGETED-PERF(_60) primitive_broadcast_join_1 parquet / none / none 2.54 2.53 +0.56% 1.43% 0.88% 1 5
          TARGETED-PERF(_60) PERF_AGG-Q2 parquet / none / none 8.90 8.85 +0.56% 0.80% 0.81% 1 5
          TARGETED-PERF(_60) primitive_empty_build_join_1 parquet / none / none 23.47 23.39 +0.35% 0.24% 0.30% 1 5
          TARGETED-PERF(_60) primitive_filter_bigint_selective parquet / none / none 0.17 0.17 +0.35% 0.91% 1.37% 1 5
          TARGETED-PERF(_60) primitive_orderby_all parquet / none / none 32.74 32.66 +0.24% 0.30% 0.24% 1 5
          TARGETED-PERF(_60) PERF_AGG-Q5 parquet / none / none 0.63 0.63 +0.02% 0.42% 0.79% 1 5
          TARGETED-PERF(_60) primitive_filter_string_selective parquet / none / none 1.94 1.94 -0.08% 2.31% 1.13% 1 5
          TARGETED-PERF(_60) primitive_filter_decimal_selective parquet / none / none 1.63 1.63 -0.10% 0.40% 0.64% 1 5
          TARGETED-PERF(_60) PERF_STRING-Q5 parquet / none / none 4.63 4.64 -0.13% 0.54% 0.67% 1 5
          TARGETED-PERF(_60) primitive_conjunct_ordering_2 parquet / none / none 9.98 10.03 -0.44% 0.41% 0.64% 1 5
          TARGETED-PERF(_60) PERF_AGG-Q7 parquet / none / none 2.04 2.05 -0.59% 2.70% 2.54% 1 5
          TARGETED-PERF(_60) primitive_orderby_bigint parquet / none / none 5.76 5.81 -0.75% 0.89% 0.40% 1 5
          TARGETED-PERF(_60) primitive_filter_in_predicate parquet / none / none 2.26 2.27 -0.84% 1.72% 1.91% 1 5
          TARGETED-PERF(_60) primitive_exchange_broadcast parquet / none / none 37.21 37.57 -0.95% 1.86% 1.55% 1 5
          TARGETED-PERF(_60) PERF_STRING-Q1 parquet / none / none 3.15 3.20 -1.33% 0.66% 0.41% 1 5
          TARGETED-PERF(_60) primitive_groupby_decimal_lowndv.test parquet / none / none 3.82 3.87 -1.34% 1.12% 1.45% 1 5
          TARGETED-PERF(_60) primitive_shuffle_join_one_to_many_string_with_groupby parquet / none / none 262.47 266.28 -1.43% 0.83% 0.65% 1 5
          TARGETED-PERF(_60) primitive_conjunct_ordering_1 parquet / none / none 0.07 0.07 -1.76% 0.46% 3.42% 1 5
          TARGETED-PERF(_60) primitive_groupby_bigint_highndv parquet / none / none 32.43 33.08 -1.97% 0.55% 0.89% 1 5
          TARGETED-PERF(_60) primitive_groupby_bigint_lowndv parquet / none / none 3.80 3.89 -2.25% 0.50% 0.88% 1 5
          TARGETED-PERF(_60) primitive_groupby_bigint_pk parquet / none / none 111.05 113.86 -2.47% 0.31% 0.25% 1 5
          TARGETED-PERF(_60) primitive_shuffle_join_union_all_with_groupby parquet / none / none 52.19 53.59 -2.60% 0.12% 0.39% 1 5
          TARGETED-PERF(_60) primitive_filter_decimal_non_selective parquet / none / none 1.64 1.69 -2.77% 1.37% 3.05% 1 5
          TARGETED-PERF(_60) PERF_AGG-Q3 parquet / none / none 12.34 12.90 -4.32% 0.81% 1.05% 1 5
          TARGETED-PERF(_60) primitive_exchange_shuffle parquet / none / none 80.49 85.96 -6.37% 3.41% 0.54% 1 5
          TARGETED-PERF(_60) PERF_AGG-Q6 parquet / none / none 2.01 2.27 -11.49% 1.05%
          • 15.97% *
          1 5
          TARGETED-PERF(_60) PERF_AGG-Q4 parquet / none / none 17.69 21.46 -17.59% 0.44% 0.10% 1 5
          TARGETED-PERF(_60) primitive_groupby_decimal_highndv parquet / none / none 21.50 31.83 I -32.43% 2.50% 0.56% 1 5

          ------------------------------------------------------------------------------------------------------------------------------------------------------------------+

          (R) Regression: TARGETED-PERF(_60) PERF_LIMIT-Q1 [parquet / none / none] (0.01s -> 0.01s [+22.95%])
          -------------------------------------------------------------------------------------------+

          Operator % of Query Avg Base Avg Delta(Avg) StdDev(%) Max Base Max Delta(Max) #Hosts #Rows Est #Rows

          -------------------------------------------------------------------------------------------+
          -------------------------------------------------------------------------------------------+

          (I) Improvement: TARGETED-PERF(_60) primitive_groupby_decimal_highndv [parquet / none / none] (31.83s -> 21.50s [-32.43%])
          -----------------------------------------------------------------------------------------------------------+

          Operator % of Query Avg Base Avg Delta(Avg) StdDev(%) Max Base Max Delta(Max) #Hosts #Rows Est #Rows

          -----------------------------------------------------------------------------------------------------------+

          01:AGGREGATE 94.77% 20.36s 30.60s -33.47% 2.53% 21.20s 30.73s -31.02% 1 3.17M 2.98M
          00:SCAN HDFS 3.65% 783.14ms 841.47ms -6.93% 3.69% 832.41ms 861.74ms -3.40% 1 360.01M 360.01M

          -----------------------------------------------------------------------------------------------------------+

          (V) Significant Variability: TARGETED-PERF(_60) primitive_conjunct_ordering_4 [parquet / none / none] (1.76% -> 12.81%)
          -----------------------------------------------------------------------------------+

          Operator % of Query StdDev(%) Base StdDev(%) Delta(StdDev(%)) #Hosts #Rows Est #Rows

          -----------------------------------------------------------------------------------+

          03:AGGREGATE 10.76% 18.81% 16.87% +11.48% 1 0 1
          01:AGGREGATE 5.02% 38.41% 14.69% +161.46% 1 0 1

          -----------------------------------------------------------------------------------+

          Significant perf change detected

          Run Description: "Base: 68f32e52bc42bef578330a4fe0edc5b292891eea vs Ref: f39d69bcd8bdc7d6d4fb42ef19966a26dea3a29d"

          Cluster Name: UNKNOWN
          Lab Run Info: UNKNOWN
          Impala Version: impalad version 2.9.0-SNAPSHOT RELEASE ()
          Baseline Impala Version: impalad version 2.9.0-SNAPSHOT RELEASE (2017-04-06)

          -------------------------------------------------------------------------+

          Workload File Format Avg (s) Delta(Avg) GeoMean(s) Delta(GeoMean)

          -------------------------------------------------------------------------+

          TPCH(_60) parquet / none / none 17.50 -1.88% 12.18 -1.70%

          -------------------------------------------------------------------------+

          ----------------------------------------------------------------------------------------------------------+

          Workload Query File Format Avg(s) Base Avg(s) Delta(Avg) StdDev(%) Base StdDev(%) Num Clients Iters

          ----------------------------------------------------------------------------------------------------------+

          TPCH(_60) TPCH-Q12 parquet / none / none 9.63 9.31 +3.50% 4.84% 2.62% 1 5
          TPCH(_60) TPCH-Q14 parquet / none / none 8.20 7.96 +2.99% 5.30% 0.30% 1 5
          TPCH(_60) TPCH-Q17 parquet / none / none 16.61 16.19 +2.64% 1.10% 0.55% 1 5
          TPCH(_60) TPCH-Q5 parquet / none / none 9.77 9.70 +0.76% 0.39% 0.71% 1 5
          TPCH(_60) TPCH-Q1 parquet / none / none 28.02 27.86 +0.59% 0.61% 0.55% 1 5
          TPCH(_60) TPCH-Q20 parquet / none / none 7.83 7.79 +0.45% 1.61% 1.53% 1 5
          TPCH(_60) TPCH-Q15 parquet / none / none 11.02 10.97 +0.42% 0.19% 0.53% 1 5
          TPCH(_60) TPCH-Q6 parquet / none / none 4.61 4.59 +0.40% 0.44% 0.98% 1 5
          TPCH(_60) TPCH-Q21 parquet / none / none 68.05 68.11 -0.09% 0.08% 0.21% 1 5
          TPCH(_60) TPCH-Q18 parquet / none / none 47.03 47.10 -0.15% 1.63% 0.28% 1 5
          TPCH(_60) TPCH-Q11 parquet / none / none 2.43 2.44 -0.58% 4.45% 2.23% 1 5
          TPCH(_60) TPCH-Q8 parquet / none / none 12.31 12.43 -0.94% 0.79% 0.54% 1 5
          TPCH(_60) TPCH-Q4 parquet / none / none 7.94 8.04 -1.16% 0.92% 0.52% 1 5
          TPCH(_60) TPCH-Q19 parquet / none / none 11.02 11.20 -1.67% 0.70% 0.39% 1 5
          TPCH(_60) TPCH-Q2 parquet / none / none 3.28 3.34 -1.95% 0.86% 2.08% 1 5
          TPCH(_60) TPCH-Q9 parquet / none / none 35.53 36.37 -2.29% 0.48% 0.23% 1 5
          TPCH(_60) TPCH-Q3 parquet / none / none 12.56 13.11 -4.19% 0.99% 0.81% 1 5
          TPCH(_60) TPCH-Q16 parquet / none / none 5.28 5.52 -4.31% 0.69% 0.49% 1 5
          TPCH(_60) TPCH-Q10 parquet / none / none 13.85 14.55 -4.82% 0.66% 0.81% 1 5
          TPCH(_60) TPCH-Q7 parquet / none / none 41.17 44.36 -7.20% 1.40% 0.29% 1 5
          TPCH(_60) TPCH-Q13 parquet / none / none 22.26 24.06 -7.46% 0.55% 0.27% 1 5
          TPCH(_60) TPCH-Q22 parquet / none / none 6.57 7.35 -10.63% 0.62% 0.50% 1 5

          ----------------------------------------------------------------------------------------------------------+

          Change-Id: If84b46a46efed9aee6af41b5f10bf3f4b15889b8
          Reviewed-on: http://gerrit.cloudera.org:8080/6687
          Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
          Tested-by: Impala Public Jenkins

          Show
          tarmstrong Tim Armstrong added a comment - IMPALA-5229 : huge page-backed buffers with TCMalloc This commit enables transparent huge pages when we're allocating via malloc(), not just mmap(). This gives us the perf benefits of huge pages, without the challenge that the mmap() path presented - the overhead of mapping and unmapping memory and the difficulty in reasoning about peak virtual memory consumption. Also sneak in some cleanup - use Rvalue refs for BufferHandle methods where appropriate. Testing: Updated backend tests to ensure this combination is covered. Ran some end-to-end tests and stress tests on my buffer pool dev branch and all looks good. Perf: Compared to current master, this provides a pretty clear perf benefit: I ran benchmarks on a single daemon with a reasonably large TPC-H scale factor. Large aggregations are much faster and everything else is the same (within variance) or slightly faster. Report Generated on 2017-04-18 Run Description: "Base: 68f32e52bc42bef578330a4fe0edc5b292891eea vs Ref: f39d69bcd8bdc7d6d4fb42ef19966a26dea3a29d" Cluster Name: UNKNOWN Lab Run Info: UNKNOWN Impala Version: impalad version 2.9.0-SNAPSHOT RELEASE () Baseline Impala Version: impalad version 2.9.0-SNAPSHOT RELEASE (2017-04-06) ------------------- --------------------- ------- ---------- ---------- ---------------+ Workload File Format Avg (s) Delta(Avg) GeoMean(s) Delta(GeoMean) ------------------- --------------------- ------- ---------- ---------- ---------------+ TARGETED-PERF(_60) parquet / none / none 19.30 -3.05% 4.91 -0.91% ------------------- --------------------- ------- ---------- ---------- ---------------+ ------------------- ------------------------------------------------------ --------------------- ------ ----------- ---------- ---------- -------------- ----------- ------+ Workload Query File Format Avg(s) Base Avg(s) Delta(Avg) StdDev(%) Base StdDev(%) Num Clients Iters ------------------- ------------------------------------------------------ --------------------- ------ ----------- ---------- ---------- -------------- ----------- ------+ TARGETED-PERF(_60) PERF_LIMIT-Q1 parquet / none / none 0.01 0.01 R +22.95% 6.12% 2.30% 1 5 TARGETED-PERF(_60) primitive_topn_bigint parquet / none / none 5.14 4.66 +10.35% 5.00% 13.39% * 1 5 TARGETED-PERF(_60) primitive_conjunct_ordering_4 parquet / none / none 0.24 0.23 +8.12% 12.81% * 1.76% 1 5 TARGETED-PERF(_60) primitive_broadcast_join_3 parquet / none / none 7.86 7.39 +6.44% 1.49% 1.41% 1 5 TARGETED-PERF(_60) PERF_STRING-Q6 parquet / none / none 10.53 10.30 +2.24% 0.61% 0.30% 1 5 TARGETED-PERF(_60) primitive_conjunct_ordering_5 parquet / none / none 17.23 16.90 +1.90% 1.61% 1.05% 1 5 TARGETED-PERF(_60) primitive_conjunct_ordering_3 parquet / none / none 3.19 3.13 +1.81% 1.47% 0.45% 1 5 TARGETED-PERF(_60) primitive_filter_bigint_non_selective parquet / none / none 1.08 1.06 +1.60% 0.26% 2.07% 1 5 TARGETED-PERF(_60) PERF_STRING-Q3 parquet / none / none 3.75 3.71 +1.14% 0.39% 0.80% 1 5 TARGETED-PERF(_60) primitive_broadcast_join_2 parquet / none / none 5.15 5.09 +1.11% 1.18% 0.89% 1 5 TARGETED-PERF(_60) PERF_STRING-Q2 parquet / none / none 3.47 3.44 +1.03% 1.27% 0.61% 1 5 TARGETED-PERF(_60) PERF_AGG-Q1 parquet / none / none 2.53 2.51 +1.01% 1.75% 1.91% 1 5 TARGETED-PERF(_60) PERF_STRING-Q7 parquet / none / none 8.37 8.31 +0.81% 0.49% 0.58% 1 5 TARGETED-PERF(_60) primitive_filter_string_non_selective parquet / none / none 1.90 1.88 +0.74% 1.81% 0.73% 1 5 TARGETED-PERF(_60) primitive_filter_string_like parquet / none / none 14.82 14.73 +0.62% 0.17% 0.02% 1 5 TARGETED-PERF(_60) primitive_top-n_all parquet / none / none 40.93 40.69 +0.61% 0.09% 0.15% 1 5 TARGETED-PERF(_60) PERF_STRING-Q4 parquet / none / none 3.76 3.74 +0.60% 0.79% 0.53% 1 5 TARGETED-PERF(_60) primitive_broadcast_join_1 parquet / none / none 2.54 2.53 +0.56% 1.43% 0.88% 1 5 TARGETED-PERF(_60) PERF_AGG-Q2 parquet / none / none 8.90 8.85 +0.56% 0.80% 0.81% 1 5 TARGETED-PERF(_60) primitive_empty_build_join_1 parquet / none / none 23.47 23.39 +0.35% 0.24% 0.30% 1 5 TARGETED-PERF(_60) primitive_filter_bigint_selective parquet / none / none 0.17 0.17 +0.35% 0.91% 1.37% 1 5 TARGETED-PERF(_60) primitive_orderby_all parquet / none / none 32.74 32.66 +0.24% 0.30% 0.24% 1 5 TARGETED-PERF(_60) PERF_AGG-Q5 parquet / none / none 0.63 0.63 +0.02% 0.42% 0.79% 1 5 TARGETED-PERF(_60) primitive_filter_string_selective parquet / none / none 1.94 1.94 -0.08% 2.31% 1.13% 1 5 TARGETED-PERF(_60) primitive_filter_decimal_selective parquet / none / none 1.63 1.63 -0.10% 0.40% 0.64% 1 5 TARGETED-PERF(_60) PERF_STRING-Q5 parquet / none / none 4.63 4.64 -0.13% 0.54% 0.67% 1 5 TARGETED-PERF(_60) primitive_conjunct_ordering_2 parquet / none / none 9.98 10.03 -0.44% 0.41% 0.64% 1 5 TARGETED-PERF(_60) PERF_AGG-Q7 parquet / none / none 2.04 2.05 -0.59% 2.70% 2.54% 1 5 TARGETED-PERF(_60) primitive_orderby_bigint parquet / none / none 5.76 5.81 -0.75% 0.89% 0.40% 1 5 TARGETED-PERF(_60) primitive_filter_in_predicate parquet / none / none 2.26 2.27 -0.84% 1.72% 1.91% 1 5 TARGETED-PERF(_60) primitive_exchange_broadcast parquet / none / none 37.21 37.57 -0.95% 1.86% 1.55% 1 5 TARGETED-PERF(_60) PERF_STRING-Q1 parquet / none / none 3.15 3.20 -1.33% 0.66% 0.41% 1 5 TARGETED-PERF(_60) primitive_groupby_decimal_lowndv.test parquet / none / none 3.82 3.87 -1.34% 1.12% 1.45% 1 5 TARGETED-PERF(_60) primitive_shuffle_join_one_to_many_string_with_groupby parquet / none / none 262.47 266.28 -1.43% 0.83% 0.65% 1 5 TARGETED-PERF(_60) primitive_conjunct_ordering_1 parquet / none / none 0.07 0.07 -1.76% 0.46% 3.42% 1 5 TARGETED-PERF(_60) primitive_groupby_bigint_highndv parquet / none / none 32.43 33.08 -1.97% 0.55% 0.89% 1 5 TARGETED-PERF(_60) primitive_groupby_bigint_lowndv parquet / none / none 3.80 3.89 -2.25% 0.50% 0.88% 1 5 TARGETED-PERF(_60) primitive_groupby_bigint_pk parquet / none / none 111.05 113.86 -2.47% 0.31% 0.25% 1 5 TARGETED-PERF(_60) primitive_shuffle_join_union_all_with_groupby parquet / none / none 52.19 53.59 -2.60% 0.12% 0.39% 1 5 TARGETED-PERF(_60) primitive_filter_decimal_non_selective parquet / none / none 1.64 1.69 -2.77% 1.37% 3.05% 1 5 TARGETED-PERF(_60) PERF_AGG-Q3 parquet / none / none 12.34 12.90 -4.32% 0.81% 1.05% 1 5 TARGETED-PERF(_60) primitive_exchange_shuffle parquet / none / none 80.49 85.96 -6.37% 3.41% 0.54% 1 5 TARGETED-PERF(_60) PERF_AGG-Q6 parquet / none / none 2.01 2.27 -11.49% 1.05% 15.97% * 1 5 TARGETED-PERF(_60) PERF_AGG-Q4 parquet / none / none 17.69 21.46 -17.59% 0.44% 0.10% 1 5 TARGETED-PERF(_60) primitive_groupby_decimal_highndv parquet / none / none 21.50 31.83 I -32.43% 2.50% 0.56% 1 5 ------------------- ------------------------------------------------------ --------------------- ------ ----------- ---------- ---------- -------------- ----------- ------+ (R) Regression: TARGETED-PERF(_60) PERF_LIMIT-Q1 [parquet / none / none] (0.01s -> 0.01s [+22.95%] ) --------- ---------- --- -------- ---------- --------- --- -------- ---------- ------ ----- ----------+ Operator % of Query Avg Base Avg Delta(Avg) StdDev(%) Max Base Max Delta(Max) #Hosts #Rows Est #Rows --------- ---------- --- -------- ---------- --------- --- -------- ---------- ------ ----- ----------+ --------- ---------- --- -------- ---------- --------- --- -------- ---------- ------ ----- ----------+ (I) Improvement: TARGETED-PERF(_60) primitive_groupby_decimal_highndv [parquet / none / none] (31.83s -> 21.50s [-32.43%] ) ------------- ---------- -------- -------- ---------- --------- -------- -------- ---------- ------ ------- ----------+ Operator % of Query Avg Base Avg Delta(Avg) StdDev(%) Max Base Max Delta(Max) #Hosts #Rows Est #Rows ------------- ---------- -------- -------- ---------- --------- -------- -------- ---------- ------ ------- ----------+ 01:AGGREGATE 94.77% 20.36s 30.60s -33.47% 2.53% 21.20s 30.73s -31.02% 1 3.17M 2.98M 00:SCAN HDFS 3.65% 783.14ms 841.47ms -6.93% 3.69% 832.41ms 861.74ms -3.40% 1 360.01M 360.01M ------------- ---------- -------- -------- ---------- --------- -------- -------- ---------- ------ ------- ----------+ (V) Significant Variability: TARGETED-PERF(_60) primitive_conjunct_ordering_4 [parquet / none / none] (1.76% -> 12.81%) ------------- ---------- --------- -------------- ---------------- ------ ----- ----------+ Operator % of Query StdDev(%) Base StdDev(%) Delta(StdDev(%)) #Hosts #Rows Est #Rows ------------- ---------- --------- -------------- ---------------- ------ ----- ----------+ 03:AGGREGATE 10.76% 18.81% 16.87% +11.48% 1 0 1 01:AGGREGATE 5.02% 38.41% 14.69% +161.46% 1 0 1 ------------- ---------- --------- -------------- ---------------- ------ ----- ----------+ Significant perf change detected Run Description: "Base: 68f32e52bc42bef578330a4fe0edc5b292891eea vs Ref: f39d69bcd8bdc7d6d4fb42ef19966a26dea3a29d" Cluster Name: UNKNOWN Lab Run Info: UNKNOWN Impala Version: impalad version 2.9.0-SNAPSHOT RELEASE () Baseline Impala Version: impalad version 2.9.0-SNAPSHOT RELEASE (2017-04-06) ---------- --------------------- ------- ---------- ---------- ---------------+ Workload File Format Avg (s) Delta(Avg) GeoMean(s) Delta(GeoMean) ---------- --------------------- ------- ---------- ---------- ---------------+ TPCH(_60) parquet / none / none 17.50 -1.88% 12.18 -1.70% ---------- --------------------- ------- ---------- ---------- ---------------+ ---------- -------- --------------------- ------ ----------- ---------- --------- -------------- ----------- ------+ Workload Query File Format Avg(s) Base Avg(s) Delta(Avg) StdDev(%) Base StdDev(%) Num Clients Iters ---------- -------- --------------------- ------ ----------- ---------- --------- -------------- ----------- ------+ TPCH(_60) TPCH-Q12 parquet / none / none 9.63 9.31 +3.50% 4.84% 2.62% 1 5 TPCH(_60) TPCH-Q14 parquet / none / none 8.20 7.96 +2.99% 5.30% 0.30% 1 5 TPCH(_60) TPCH-Q17 parquet / none / none 16.61 16.19 +2.64% 1.10% 0.55% 1 5 TPCH(_60) TPCH-Q5 parquet / none / none 9.77 9.70 +0.76% 0.39% 0.71% 1 5 TPCH(_60) TPCH-Q1 parquet / none / none 28.02 27.86 +0.59% 0.61% 0.55% 1 5 TPCH(_60) TPCH-Q20 parquet / none / none 7.83 7.79 +0.45% 1.61% 1.53% 1 5 TPCH(_60) TPCH-Q15 parquet / none / none 11.02 10.97 +0.42% 0.19% 0.53% 1 5 TPCH(_60) TPCH-Q6 parquet / none / none 4.61 4.59 +0.40% 0.44% 0.98% 1 5 TPCH(_60) TPCH-Q21 parquet / none / none 68.05 68.11 -0.09% 0.08% 0.21% 1 5 TPCH(_60) TPCH-Q18 parquet / none / none 47.03 47.10 -0.15% 1.63% 0.28% 1 5 TPCH(_60) TPCH-Q11 parquet / none / none 2.43 2.44 -0.58% 4.45% 2.23% 1 5 TPCH(_60) TPCH-Q8 parquet / none / none 12.31 12.43 -0.94% 0.79% 0.54% 1 5 TPCH(_60) TPCH-Q4 parquet / none / none 7.94 8.04 -1.16% 0.92% 0.52% 1 5 TPCH(_60) TPCH-Q19 parquet / none / none 11.02 11.20 -1.67% 0.70% 0.39% 1 5 TPCH(_60) TPCH-Q2 parquet / none / none 3.28 3.34 -1.95% 0.86% 2.08% 1 5 TPCH(_60) TPCH-Q9 parquet / none / none 35.53 36.37 -2.29% 0.48% 0.23% 1 5 TPCH(_60) TPCH-Q3 parquet / none / none 12.56 13.11 -4.19% 0.99% 0.81% 1 5 TPCH(_60) TPCH-Q16 parquet / none / none 5.28 5.52 -4.31% 0.69% 0.49% 1 5 TPCH(_60) TPCH-Q10 parquet / none / none 13.85 14.55 -4.82% 0.66% 0.81% 1 5 TPCH(_60) TPCH-Q7 parquet / none / none 41.17 44.36 -7.20% 1.40% 0.29% 1 5 TPCH(_60) TPCH-Q13 parquet / none / none 22.26 24.06 -7.46% 0.55% 0.27% 1 5 TPCH(_60) TPCH-Q22 parquet / none / none 6.57 7.35 -10.63% 0.62% 0.50% 1 5 ---------- -------- --------------------- ------ ----------- ---------- --------- -------------- ----------- ------+ Change-Id: If84b46a46efed9aee6af41b5f10bf3f4b15889b8 Reviewed-on: http://gerrit.cloudera.org:8080/6687 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              tarmstrong Tim Armstrong
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development