Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6033

the description abou the "spark.worker.cleanup.enabled" is not matched with the code

    Details

    • Type: Documentation
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 1.2.1
    • Fix Version/s: 1.0.3, 1.1.2, 1.2.2, 1.3.0
    • Component/s: Documentation
    • Labels:
      None

      Description

      Some error about the section Cluster Launch Scripts in the http://spark.apache.org/docs/latest/spark-standalone.html

      In the description about the property spark.worker.cleanup.enabled, it states that all the directory under the work dir will be removed whether the application is running or not.

      After checking the implementation in the code level, I found that only the stopped application dirs would be removed. So the description in the document is incorrect.

      the code implementation in worker.scala

      WorkDirCleanup
      case WorkDirCleanup =>
            // Spin up a separate thread (in a future) to do the dir cleanup; don't tie up worker actor
            val cleanupFuture = concurrent.future {
              val appDirs = workDir.listFiles()
              if (appDirs == null) {
                throw new IOException("ERROR: Failed to list files in " + appDirs)
              }
              appDirs.filter { dir =>
                // the directory is used by an application - check that the application is not running
                // when cleaning up
                val appIdFromDir = dir.getName
                val isAppStillRunning = executors.values.map(_.appId).contains(appIdFromDir)
                dir.isDirectory && !isAppStillRunning &&
                !Utils.doesDirectoryContainAnyNewFiles(dir, APP_DATA_RETENTION_SECS)
              }.foreach { dir => 
                logInfo(s"Removing directory: ${dir.getPath}")
                Utils.deleteRecursively(dir)
              }
            }
      
            cleanupFuture onFailure {
              case e: Throwable =>
                logError("App dir cleanup failed: " + e.getMessage, e)
            }
      

        Attachments

          Activity

            People

            • Assignee:
              hseagle pengxu
              Reporter:
              hseagle pengxu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: