Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3902

Multi-threaded query execution

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Reopened
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: Impala 2.6.0
    • Fix Version/s: None
    • Component/s: Backend
    • Labels:
      None

      Description

      Currently, a single query fragment is run in a quasi-single threaded manner on a node: the scanners are run in multiple threads, but all other operators (joins, aggregation) are run in the main thread.

      The goal is to add multi-threaded execution on a single node by running multiple fragment instances (each of which runs in a single thread).

        Attachments

          Issue Links

          1.
          Switch to per-query exec rpc Sub-task Resolved Marcel Kinard
          2.
          Factor out join build-side Sub-task Resolved Tim Armstrong
          3.
          Scheduler improvements for running multiple fragment instances on a single backend Sub-task Resolved Marcel Kinard
          4.
          Single-threaded scan node Sub-task Resolved Alexander Behm
          5.
          Generate codegen module without operator/query-specific constants Sub-task Resolved Michael Ho
          6.
          Introduce query-wide execution state. Sub-task Resolved Marcel Kinard
          7.
          Experimental flag for running all queries with mt_dop Sub-task Resolved Tim Armstrong
          8.
          Planner should disallow queries with mt_dop > 0 that are not executable. Sub-task Resolved Alexander Behm
          9.
          Standardize on MT-related data structures in the coordinator and scheduler Sub-task Resolved Marcel Kinard
          10.
          COMPUTE STATS on Parquet tables uses MT_DOP=4 by default Sub-task Resolved Alexander Behm
          11.
          Adjust maximum size of row batch queue for non-Parquet scans with MT_DOP>0. Sub-task Resolved Alexander Behm
          12.
          Single-threaded KuduScanNode Sub-task Resolved Joe McDonnell
          13.
          ReleaseResources() should not destroy control structures Sub-task Resolved Unassigned
          14.
          COMPUTE STATS uses MT_DOP=4 by default Sub-task Resolved Tim Armstrong
          15.
          Create tests for multi-threaded query execution Sub-task Resolved Unassigned
          16.
          Allow graceful fallback to mt_dop=0 Sub-task In Progress Tim Armstrong
          17.
          Admission control accounting for mt_dop Sub-task In Progress Tim Armstrong
          18.
          Use better algorithm for allocating scan ranges to finstances within a daemon in schedule Sub-task Resolved Tim Armstrong
          19.
          Parallelise all plans, including UNION, when mt_dop > 1 Sub-task In Progress Tim Armstrong
          20.
          Share codegen work between fragment instances Sub-task Open Bikramjeet Vig
          21.
          Fix runtime filter bugs with mt_dop Sub-task Open Unassigned
          22.
          Limit number of files generated by unpartitioned insert Sub-task Open Unassigned
          23.
          Increase runtime filter wait timeout for mt_dop Sub-task Open Unassigned
          24.
          Aggregate runtime filters locally Sub-task Open Unassigned
          25.
          MT Scanners do not check runtime filters per-file before processing each split Sub-task Open Unassigned
          26.
          Consider adding local aggregation step for mt_dop plans Sub-task Open Unassigned
          27.
          Add backend support for join build sinks in parallel plans Sub-task Open Unassigned
          28.
          Switch default to multi-threaded query execution Sub-task Open Unassigned
          29.
          Fix all the TODO-MT comments Sub-task Open Unassigned
          30.
          Update planner decisions to factor in mt_dop Sub-task Open Unassigned

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                marcelk Marcel Kinard
              • Votes:
                4 Vote for this issue
                Watchers:
                28 Start watching this issue

                Dates

                • Created:
                  Updated: