Details

      Description

      The execution-related rpcs in ImpalaInternalService (ExecPlanFragment, CancelPlanFragment) are currently fragment-instance oriented. This is very inefficient because a) starting up a single query involves lots of rpcs, b) it makes it hard to share query-wide data structures such as the descriptor table.

      Instead, let's move to a per-query focus and replace the existing "execute" and "cancel" rpcs with per-query equivalents.

        Issue Links

          Activity

          Hide
          henryr Henry Robinson added a comment -

          The descriptor table redundancy can have a big impact (IMPALA-2560). Moving to a batch execution model will at least amortise the cost of the descriptor tables across all fragments. It won't be perfect in the case where scans don't have the same degree of parallelism (because some backends won't need all the table descriptors), but should get us a long way.

          Show
          henryr Henry Robinson added a comment - The descriptor table redundancy can have a big impact ( IMPALA-2560 ). Moving to a batch execution model will at least amortise the cost of the descriptor tables across all fragments. It won't be perfect in the case where scans don't have the same degree of parallelism (because some backends won't need all the table descriptors), but should get us a long way.
          Hide
          marcelk Marcel Kornacker added a comment -

          commit 368115cdae98344e5c826f099582a60aa536951d
          Author: Marcel Kornacker <marcel@cloudera.com>
          Date: Mon Oct 31 14:42:51 2016 -0700

          IMPALA-2550: Switch to per-query exec rpc

          Coordinator:

          • FragmentInstanceState -> BackendState, which in turn records
            FragmentInstanceStats

          QueryState

          • does query-wide setup in a separate thread (which also launches
            the instance exec threads)
          • has a query-wide 'prepared' state at which point all static setup
            is done and all FragmentInstanceStates are accessible

          Also renamed QueryExecState to ClientRequestState.

          Simplified handling of execution status (in FragmentInstanceState):

          • status only transmitted via ReportExecStatus rpc
          • in particular, it's not returned anymore from the Cancel rpc

          FIS: Fixed bugs related to partially-prepared state (in Close() and ReleaseThreadToken())

          Change-Id: I20769e420711737b6b385c744cef4851cee3facd
          Reviewed-on: http://gerrit.cloudera.org:8080/6535
          Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
          Tested-by: Impala Public Jenkins

          Show
          marcelk Marcel Kornacker added a comment - commit 368115cdae98344e5c826f099582a60aa536951d Author: Marcel Kornacker <marcel@cloudera.com> Date: Mon Oct 31 14:42:51 2016 -0700 IMPALA-2550 : Switch to per-query exec rpc Coordinator: FragmentInstanceState -> BackendState, which in turn records FragmentInstanceStats QueryState does query-wide setup in a separate thread (which also launches the instance exec threads) has a query-wide 'prepared' state at which point all static setup is done and all FragmentInstanceStates are accessible Also renamed QueryExecState to ClientRequestState. Simplified handling of execution status (in FragmentInstanceState): status only transmitted via ReportExecStatus rpc in particular, it's not returned anymore from the Cancel rpc FIS: Fixed bugs related to partially-prepared state (in Close() and ReleaseThreadToken()) Change-Id: I20769e420711737b6b385c744cef4851cee3facd Reviewed-on: http://gerrit.cloudera.org:8080/6535 Reviewed-by: Marcel Kornacker <marcel@cloudera.com> Tested-by: Impala Public Jenkins

            People

            • Assignee:
              marcelk Marcel Kornacker
              Reporter:
              henryr Henry Robinson
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development