Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4514

SparkContext localProperties does not inherit property updates across thread reuse

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.1.0, 1.1.1, 1.2.0
    • Fix Version/s: 2.0.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      The current job group id of a Spark context is stored in the localProperties member value. This data structure is designed to be thread local, and its settings are not preserved when ComplexFutureAction instantiates a new Future.

      One consequence of this is that takeAsync() does not behave in the same way as other async actions, e.g. countAsync(). For example, this test (if copied into StatusTrackerSuite.scala), will fail, because "my-job-group2" is not propagated to the Future which actually instantiates the job:

        test("getJobIdsForGroup() with takeAsync()") {
          sc = new SparkContext("local", "test", new SparkConf(false))
          sc.setJobGroup("my-job-group2", "description")
          sc.statusTracker.getJobIdsForGroup("my-job-group2") should be (Seq.empty)
          val firstJobFuture = sc.parallelize(1 to 1000, 1).takeAsync(1)
          val firstJobId = eventually(timeout(10 seconds)) {
            firstJobFuture.jobIds.head
          }
          eventually(timeout(10 seconds)) {
            sc.statusTracker.getJobIdsForGroup("my-job-group2") should be (Seq(firstJobId))
          }
        }
      

      It also impacts current PR for SPARK-1021, which involves additional uses of ComplexFutureAction.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                reggert1980 Richard W. Eggert II
                Reporter:
                eje Erik Erlandson
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: