[SPARK-29105] SHS may delete driver log file of in progress application

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 17/Sep/19 00:02
• Updated: 18/Sep/19 16:12
• Resolved: 18/Sep/19 16:12

Description

There's an issue with how the SHS cleans driver logs that is similar to the problem of event logs: because the file size is not updated when you write to it, the SHS fails to detect activity and thus may delete the file while it's still being written to.

SPARK-24787 added a workaround in the SHS so that it can detect that situation for in-progress apps, replacing the previous solution which was too slow for event logs.

But that doesn't work for driver logs because they do not follow the same pattern (different file names for in-progress files), and thus would require the SHS to open the driver log files on every scan, which is expensive.

The old approach (using the hsync API) seems to be a good match for the driver logs, though, which don't slow down the listener bus like event logs do.

Issue Links

• links to: GitHub Pull Request #25819