[SPARK-2261] Spark application event logs are not very NameNode-friendly

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.3.0
• Component/s: Spark Core
• Target Version/s: 1.3.0
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 24/Jun/14 20:38
• Updated: 21/Dec/14 10:25
• Resolved: 20/Dec/14 02:26

Description

Currently, EventLoggingListener will generate application logs using, in the worst case, five different entries in the file system:

• The directory to hold the files
• One file for the Spark version
• One file for the event logs
• One file to identify the compression codec of the event logs
• One file to say the application is finished.

It would be better to be more friendly to the NameNodes and use a single entry for all of those.

Issue Links

• links to: [Github] Pull Request #1222 (vanzin)