Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3902 Multi-threaded query execution
  3. IMPALA-4577

Adjust maximum size of row batch queue for non-Parquet scans with MT_DOP>0.

    Details

      Description

      When MT_DOP is set, non-Parquet scans are run with the old scan node with a fixed NUM_SCANNER_THREADS=1

      We should adjust the maximum size of the row batch queue based on MT_DOP for each such scan instance to avoid a significant increase in
      the memory consumption of such scans.

        Activity

        Hide
        alex.behm Alexander Behm added a comment -

        commit b237d1316eac6ccde4ce395b4c238997be3e2f8b
        Author: Alex Behm <alex.behm@cloudera.com>
        Date: Fri Dec 2 10:17:22 2016 -0800

        IMPALA-4577: Adjust maximum size of row batch queue with MT_DOP.

        When MT_DOP is set, non-Parquet scans are run with the old
        scan node with a fixed NUM_SCANNER_THREADS=1. This patch
        adjust the maximum size of the row batch queue based on MT_DOP
        for each such scan instance to avoid a significant increase in
        the memory consumption of such scans. The max queued batches
        per scan-node instance is at least 2 to always allow for some
        parallelism between the producer/consumer.

        Decreases the maximum allowed value for MT_DOP from 128 to 64.

        Change-Id: Ic2aa260f9265ec21173fb703c41934964ece6485
        Reviewed-on: http://gerrit.cloudera.org:8080/5330
        Reviewed-by: Alex Behm <alex.behm@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        alex.behm Alexander Behm added a comment - commit b237d1316eac6ccde4ce395b4c238997be3e2f8b Author: Alex Behm <alex.behm@cloudera.com> Date: Fri Dec 2 10:17:22 2016 -0800 IMPALA-4577 : Adjust maximum size of row batch queue with MT_DOP. When MT_DOP is set, non-Parquet scans are run with the old scan node with a fixed NUM_SCANNER_THREADS=1. This patch adjust the maximum size of the row batch queue based on MT_DOP for each such scan instance to avoid a significant increase in the memory consumption of such scans. The max queued batches per scan-node instance is at least 2 to always allow for some parallelism between the producer/consumer. Decreases the maximum allowed value for MT_DOP from 128 to 64. Change-Id: Ic2aa260f9265ec21173fb703c41934964ece6485 Reviewed-on: http://gerrit.cloudera.org:8080/5330 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Internal Jenkins

          People

          • Assignee:
            alex.behm Alexander Behm
            Reporter:
            alex.behm Alexander Behm
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development