[SPARK-37659] Fix FsHistoryProvider race condition between list and delet log info

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.2, 3.2.1, 3.3.0
• Fix Version/s: 3.2.1, 3.3.0
• Component/s: Web UI
• Assignee: XiDuo You
• Reporter: XiDuo You
• Created: 16/Dec/21 06:13
• Updated: 02/May/22 08:12
• Resolved: 23/Dec/21 07:16

Description

After SPARK-29043, FsHistoryProvider will list the log info without waitting all `mergeApplicationListing` task finished.

However the `LevelDBIterator` of list log info is not thread safe if some other threads delete the related log info at same time.

There is the error msg:

21/12/15 14:12:02 ERROR FsHistoryProvider: Exception in checking for event log updates
java.util.NoSuchElementException: 1^@__main__^@+hdfs://xxx/application_xxx.inprogress
        at org.apache.spark.util.kvstore.LevelDB.get(LevelDB.java:132)
        at org.apache.spark.util.kvstore.LevelDBIterator.next(LevelDBIterator.java:137)
        at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44)
        at scala.collection.Iterator.foreach(Iterator.scala:941)
        at scala.collection.Iterator.foreach$(Iterator.scala:941)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
        at scala.collection.IterableLike.foreach(IterableLike.scala:74)
        at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
        at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
        at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
        at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:184)
        at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:47)
        at scala.collection.TraversableLike.to(TraversableLike.scala:678)
        at scala.collection.TraversableLike.to$(TraversableLike.scala:675)
        at scala.collection.AbstractTraversable.to(Traversable.scala:108)
        at scala.collection.TraversableOnce.toList(TraversableOnce.scala:299)
        at scala.collection.TraversableOnce.toList$(TraversableOnce.scala:299)
        at scala.collection.AbstractTraversable.toList(Traversable.scala:108)
        at org.apache.spark.deploy.history.FsHistoryProvider.checkForLogs(FsHistoryProvider.scala:588)
        at org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$startPolling$3(FsHistoryProvider.scala:299)

Issue Links

• is cloned by: SPARK-39082 Fix FsHistoryProvider race condition between update and clean app data
• is related to: SPARK-39083 Fix FsHistoryProvider race condition between update and clean app data
• links to: [Github] Pull Request #34919 (ulysses-you)
• links to: [Github] Pull Request #35003 (ulysses-you)