[SPARK-26186] In progress applications with last updated time is lesser than the cleaning interval are getting removed during cleaning logs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0, 3.0.0
• Fix Version/s: 2.4.1, 3.0.0
• Component/s: Spark Core
• Assignee: shahid
• Reporter: shahid
• Created: 27/Nov/18 17:56
• Updated: 29/Nov/18 17:52
• Resolved: 29/Nov/18 17:52

Description

Inporgress applications with last updated time is withing the cleaning interval are getting deleted.

 

Added a UT to test the scenario.

test("should not clean inprogress application with lastUpdated time less the maxTime") {
    val firstFileModifiedTime = TimeUnit.DAYS.toMillis(1)
    val secondFileModifiedTime = TimeUnit.DAYS.toMillis(6)
    val maxAge = TimeUnit.DAYS.toMillis(7)
    val clock = new ManualClock(0)
    val provider = new FsHistoryProvider(
      createTestConf().set("spark.history.fs.cleaner.maxAge", s"${maxAge}ms"), clock)
    val log = newLogFile("inProgressApp1", None, inProgress = true)
    writeFile(log, true, None,
      SparkListenerApplicationStart(
        "inProgressApp1", Some("inProgressApp1"), 3L, "test", Some("attempt1"))
    )
    clock.setTime(firstFileModifiedTime)
    provider.checkForLogs()
    writeFile(log, true, None,
      SparkListenerApplicationStart(
        "inProgressApp1", Some("inProgressApp1"), 3L, "test", Some("attempt1")),
      SparkListenerJobStart(0, 1L, Nil, null)
    )

    clock.setTime(secondFileModifiedTime)
    provider.checkForLogs()
    clock.setTime(TimeUnit.DAYS.toMillis(10))
    writeFile(log, true, None,
      SparkListenerApplicationStart(
        "inProgressApp1", Some("inProgressApp1"), 3L, "test", Some("attempt1")),
      SparkListenerJobStart(0, 1L, Nil, null),
      SparkListenerJobEnd(0, 1L, JobSucceeded)
    )
    provider.checkForLogs()
    // This should not trigger any cleanup
    updateAndCheck(provider) { list =>
      list.size should be(1)
    }
  }

Issue Links

• links to: [Github] Pull Request #23158 (shahidki31)
• links to: [Github] Pull Request #23158 (shahidki31)