Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47819

Use asynchronous callback for execution cleanup

    XMLWordPrintableJSON

Details

    Description

      Expired sessions are regularly checked and cleaned up by a maintenance thread. However, currently, this process is synchronous. Therefore, in rare cases, interrupting the execution thread of a query in a session can take hours, causing the entire maintenance process to stall, resulting in a large amount of memory not being cleared.

      We address this by introducing asynchronous callbacks for execution cleanup, avoiding synchronous joins of execution threads, and preventing the maintenance thread from stalling in the above scenarios. To be more specific, instead of calling runner.join() in ExecutorHolder.close(), we set a post-cleanup function as the callback through runner.processOnCompletion, which will be called asynchronously once the execution runner is completed or interrupted. In this way, the maintenance thread won't get blocked on joining an execution thread.

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            xi.lyu Xi Lyu
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: