Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5616

Hash Agg Spill: OOM while reading irregular varchar data

    Details

      Description

      An OOM while aggregating a table of two varchar columns where sizes vary significantly ( about 8 bytes long in average, but 250 bytes max )

      alter session set `planner.width.max_per_node` = 1;
      alter session set `planner.memory.max_query_memory_per_node` = 327127360;
      select count( * ) from (select max(`filename`) from dfs.`/drill/testdata/hash-agg/data2` group by no_nulls_col, nulls_col) d;

      Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.
      
      OOM at Second Phase. Partitions: 2. Estimated batch size: 12255232. Planned batches: 0. Rows spilled so far: 434127447 Memory limit: 163563680 so far allocated: 150601728.
      Fragment 1:0
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ben-zvi Boaz Ben-Zvi
                Reporter:
                ben-zvi Boaz Ben-Zvi
                Reviewer:
                Paul Rogers
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 96h
                  96h
                  Remaining:
                  Remaining Estimate - 96h
                  96h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified