Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5715

Performance of refactored HashAgg operator regressed

    Details

      Description

      When running the following simple HashAgg-based query on a TPCH-table - Lineitem with 6Billion rows on a 10 node setup (with a single partition to disable any possible spilling to disk)

      select count(*) 
      from (
        select l_quantity
          , count(l_orderkey) 
        from lineitem 
        group by l_quantity 
      )  

      the runtime increased from 7.378 sec to 11.323 sec [reported by the JDBC client].

      To disable spill-to-disk in Drill-1.11.0, the drill-override.conf was modified to

      drill.exec.hashagg.num_partitions : 1

      Attached are two profiles
      Drill 1.10.0 : 2675cc73-9481-16e0-7d21-5f1338611e5f.sys.drill
      Drill 1.11.0 : 2675de42-3789-47b8-29e8-c5077af136db.sys.drill

      A separate run was done for both scenarios with the planner.width.max_per_node=10 and profiled with YourKit.

      Image snippets are attached, indicating the hotspots in both builds:
      Drill 1.10.0 :
      Profile: 26736242-d084-6604-aac9-927e729da755.sys.drill
      CallTree: drill-1.10.0_callTree.png
      HotSpot: drill-1.10.0_hotspot.png

      Drill 1.11.0 :
      Profile: 26736615-9e86-dac9-ad77-b022fd791f67.sys.drill
      CallTree: drill-1.11.0_callTree.png
      HotSpot: drill-1.11.0_hotspot.png

        Attachments

        1. drill-1.11.0_hotspot.png
          76 kB
          Kunal Khatua
        2. drill-1.11.0_callTree.png
          131 kB
          Kunal Khatua
        3. drill-1.10.0_hotspot.png
          59 kB
          Kunal Khatua
        4. drill-1.10.0_callTree.png
          120 kB
          Kunal Khatua
        5. 2675de42-3789-47b8-29e8-c5077af136db.sys.drill
          916 kB
          Kunal Khatua
        6. 2675cc73-9481-16e0-7d21-5f1338611e5f.sys.drill
          808 kB
          Kunal Khatua
        7. 26736615-9e86-dac9-ad77-b022fd791f67.sys.drill
          409 kB
          Kunal Khatua
        8. 26736242-d084-6604-aac9-927e729da755.sys.drill
          361 kB
          Kunal Khatua

          Issue Links

            Activity

              People

              • Assignee:
                ben-zvi Boaz Ben-Zvi
                Reporter:
                kkhatua Kunal Khatua
                Reviewer:
                Paul Rogers
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: