Spark history server file cleaner excludes in-progress files

When we restart history server, it does spend a lot of time to load/replay incomplete applications which mean the inprogress log files in the log folder.

We have already enabled "spark.history.fs.cleaner.enabled" but seems like it's skipping the inprogress files.

I checked the log folder and saw that there are many old orphan files. Probably files left over due to spark-driver failures or OOMs.