Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4280 Clean up tasks in Coordinator, QueryExecState and others.
  3. IMPALA-4905

Fragments always report insert status, even if not insert query

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • Impala 2.7.0
    • Impala 2.10.0
    • Distributed Exec
    • None

    Description

      if (done) {
            TInsertExecStatus insert_status;
      
            if (runtime_state->hdfs_files_to_move()->size() > 0) {
              insert_status.__set_files_to_move(*runtime_state->hdfs_files_to_move());
            }
            if (runtime_state->per_partition_status()->size() > 0) {
              insert_status.__set_per_partition_status(*runtime_state->per_partition_status());
            }
      
            params.__set_insert_exec_status(insert_status);
          }
      

      This means that any fragment will always set insert_exec_status in its response, even if it's not an INSERT query.

      However, in the RPC handler, Coordinator::UpdateFragmentExecStatus(), we have:

      if (params.done && params.__isset.insert_exec_status) {
          lock_guard<mutex> l(lock_);
          // Merge in table update data (partitions written to, files to be moved as part of
          // finalization)
          for (const PartitionStatusMap::value_type& partition:
               params.insert_exec_status.per_partition_status) {
          // etc
      

      which means that the RPC will always try and take the query exec state lock, for every 'done' report. With lots of fragment instances, this can lead to some severe serialisation of reports when the query finishes.

      The simplest workaround is not to set insert_exec_status for SELECT queries. But a better solution (that will help INSERTs as well) is not to try and do the merge here, but instead in Coordinator::FinalizeSuccessfulInsert(), saving the TInsertExecStatus in the fragment instance state until that point.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            henryr Henry Robinson
            henryr Henry Robinson
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment