Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1346

Sorter fails with out-of-memory in queries with multiple spilling operators

    Details

      Description

      An example of this is a join that is spilling followed by a sort (or aggregate) that also spills.

      These queries use memory very ineffective causing many of them to fail. The blocking
      operator starts accumulating memory and over time, the streaming operator will have none left (it is streaming so therefore gives up memory as it processes). As streaming operator eventually has too little memory to repartition and fails.

      We need coordination within an query.

      Workaround
      Increase mem_limit for query or reduce query size (by processing only a subset of data at a time).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tarmstrong Tim Armstrong
                Reporter:
                nong_impala_60e1 Nong Li
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: