[SPARK-36354] EventLogFileReaders should not complain in case of no event log files

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.2, 3.2.0
• Fix Version/s: 3.2.0, 3.1.3
• Component/s: Spark Core
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 30/Jul/21 07:47
• Updated: 04/Aug/21 12:16
• Resolved: 04/Aug/21 12:16

Description

21/07/30 07:38:26 WARN FsHistoryProvider: Error while reading new log s3a://.../eventlog_v2_spark-95b5c736c8e44037afcf152534d08771
java.lang.IllegalArgumentException: requirement failed: Log directory must contain at least one event log file!
        at scala.Predef$.require(Predef.scala:281)
        at org.apache.spark.deploy.history.RollingEventLogFilesFileReader.files$lzycompute(EventLogFileReaders.scala:216)

$ aws s3 ls s3://.../eventlog_v2_spark-95b5c736c8e44037afcf152534d08771/
2021-06-26 22:31:40          0 appstatus_spark-95b5c736c8e44037afcf152534d08771.inprogress

Issue Links

• links to: [Github] Pull Request #33586 (dongjoon-hyun)
• links to: [Github] Pull Request #33586 (dongjoon-hyun)