[SPARK-6033] the description abou the "spark.worker.cleanup.enabled" is not matched with the code

• Type: Documentation
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.2.0, 1.2.1
• Fix Version/s: 1.0.3, 1.1.2, 1.2.2, 1.3.0
• Component/s: Documentation
• Assignee: pengxu
• Reporter: pengxu
• Created: 26/Feb/15 06:46
• Updated: 27/Feb/15 12:37
• Resolved: 27/Feb/15 12:37

Description

Some error about the section Cluster Launch Scripts in the http://spark.apache.org/docs/latest/spark-standalone.html

In the description about the property spark.worker.cleanup.enabled, it states that all the directory under the work dir will be removed whether the application is running or not.

After checking the implementation in the code level, I found that only the stopped application dirs would be removed. So the description in the document is incorrect.

the code implementation in worker.scala

WorkDirCleanup

case WorkDirCleanup =>
      // Spin up a separate thread (in a future) to do the dir cleanup; don't tie up worker actor
      val cleanupFuture = concurrent.future {
        val appDirs = workDir.listFiles()
        if (appDirs == null) {
          throw new IOException("ERROR: Failed to list files in " + appDirs)
        }
        appDirs.filter { dir =>
          // the directory is used by an application - check that the application is not running
          // when cleaning up
          val appIdFromDir = dir.getName
          val isAppStillRunning = executors.values.map(_.appId).contains(appIdFromDir)
          dir.isDirectory && !isAppStillRunning &&
          !Utils.doesDirectoryContainAnyNewFiles(dir, APP_DATA_RETENTION_SECS)
        }.foreach { dir => 
          logInfo(s"Removing directory: ${dir.getPath}")
          Utils.deleteRecursively(dir)
        }
      }

      cleanupFuture onFailure {
        case e: Throwable =>
          logError("App dir cleanup failed: " + e.getMessage, e)
      }

Issue Links

• links to: [Github] Pull Request #4803 (hseagle)