Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26186

In progress applications with last updated time is lesser than the cleaning interval are getting removed during cleaning logs

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0, 3.0.0
    • Fix Version/s: 2.4.1, 3.0.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      Inporgress applications with last updated time is withing the cleaning interval are getting deleted.

       

      Added a UT to test the scenario.

      test("should not clean inprogress application with lastUpdated time less the maxTime") {
          val firstFileModifiedTime = TimeUnit.DAYS.toMillis(1)
          val secondFileModifiedTime = TimeUnit.DAYS.toMillis(6)
          val maxAge = TimeUnit.DAYS.toMillis(7)
          val clock = new ManualClock(0)
          val provider = new FsHistoryProvider(
            createTestConf().set("spark.history.fs.cleaner.maxAge", s"${maxAge}ms"), clock)
          val log = newLogFile("inProgressApp1", None, inProgress = true)
          writeFile(log, true, None,
            SparkListenerApplicationStart(
              "inProgressApp1", Some("inProgressApp1"), 3L, "test", Some("attempt1"))
          )
          clock.setTime(firstFileModifiedTime)
          provider.checkForLogs()
          writeFile(log, true, None,
            SparkListenerApplicationStart(
              "inProgressApp1", Some("inProgressApp1"), 3L, "test", Some("attempt1")),
            SparkListenerJobStart(0, 1L, Nil, null)
          )
      
          clock.setTime(secondFileModifiedTime)
          provider.checkForLogs()
          clock.setTime(TimeUnit.DAYS.toMillis(10))
          writeFile(log, true, None,
            SparkListenerApplicationStart(
              "inProgressApp1", Some("inProgressApp1"), 3L, "test", Some("attempt1")),
            SparkListenerJobStart(0, 1L, Nil, null),
            SparkListenerJobEnd(0, 1L, JobSucceeded)
          )
          provider.checkForLogs()
          // This should not trigger any cleanup
          updateAndCheck(provider) { list =>
            list.size should be(1)
          }
        }
      

        Attachments

          Activity

            People

            • Assignee:
              shahid shahid
              Reporter:
              shahid shahid
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: