[SPARK-26914] ThriftServer scheduler pool may be unpredictably when using fair schedule mode

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: zhoukang
• Reporter: zhoukang
• Created: 18/Feb/19 11:38
• Updated: 01/Apr/19 17:17
• Resolved: 28/Mar/19 14:25

Description

When using fair scheduler mode for thrift server, we may have unpredictable result.

val pool = sessionToActivePool.get(parentSession.getSessionHandle)
    if (pool != null) {
      sqlContext.sparkContext.setLocalProperty(SparkContext.SPARK_SCHEDULER_POOL, pool)
    }

Here is an example:
We have some query will use default pool, however it submit to 'normal' pool.

I changed code and add some log.Got some strange result.

Then i found out that the localProperties of SparkContext may has unpredictable result when call setLocalProperty. And since thriftserver use thread pool to execute queries, it will trigger this bug sometimes.

/**
   * Set a local property that affects jobs submitted from this thread, such as the Spark fair
   * scheduler pool. User-defined properties may also be set here. These properties are propagated
   * through to worker tasks and can be accessed there via
   * [[org.apache.spark.TaskContext#getLocalProperty]].
   *
   * These properties are inherited by child threads spawned from this thread. This
   * may have unexpected consequences when working with thread pools. The standard java
   * implementation of thread pools have worker threads spawn other worker threads.
   * As a result, local properties may propagate unpredictably.
   */
  def setLocalProperty(key: String, value: String) {
    if (value == null) {
      localProperties.get.remove(key)
    } else {
      localProperties.get.setProperty(key, value)
    }
  }

Issue Links

• is duplicated by: SPARK-26992 Fix STS scheduler pool correct delivery
• links to: GitHub Pull Request #23826