Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6769

Usage of the ListenerBus in YarnClusterSuite is wrong

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.4.0
    • Component/s: Tests, YARN
    • Labels:
      None
    • Target Version/s:

      Description

      In YarnClusterSuite, a test case uses `SaveExecutorInfo` to handle ExecutorAddedEvent as follows.

      private class SaveExecutorInfo extends SparkListener {
        val addedExecutorInfos = mutable.Map[String, ExecutorInfo]()
      
        override def onExecutorAdded(executor: SparkListenerExecutorAdded) {
          addedExecutorInfos(executor.executorId) = executor.executorInfo
        }
      }
      
      ...
      
          listener = new SaveExecutorInfo
          val sc = new SparkContext(new SparkConf()
            .setAppName("yarn \"test app\" 'with quotes' and \\back\\slashes and $dollarSigns"))
          sc.addSparkListener(listener)
          val status = new File(args(0))
          var result = "failure"
          try {
            val data = sc.parallelize(1 to 4, 4).collect().toSet
            assert(sc.listenerBus.waitUntilEmpty(WAIT_TIMEOUT_MILLIS))
            data should be (Set(1, 2, 3, 4))
            result = "success"
          } finally {
            sc.stop()
            Files.write(result, status, UTF_8)
          }
      

      But, the usage is wrong because Executors will spawn during initializing SparkContext and SparkContext#addSparkListener should be invoked after the initialization, thus after Executors spawn, so SaveExecutorInfo cannot handle ExecutorAddedEvent.

      Following code refers the result of the handling ExecutorAddedEvent. Because of the reason above, we cannot reach the assertion.

          // verify log urls are present
          listener.addedExecutorInfos.values.foreach { info =>
            assert(info.logUrlMap.nonEmpty)
          }
      

        Attachments

          Activity

            People

            • Assignee:
              sarutak Kousuke Saruta
              Reporter:
              sarutak Kousuke Saruta
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: