[SPARK-6066] Metadata in event log makes it very difficult for external libraries to parse event log

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.3.0
• Component/s: Spark Core
• Target Version/s: 1.3.0
• Assignee: Andrew Or
• Reporter: Kay Ousterhout
• Created: 27/Feb/15 20:10
• Updated: 03/Mar/15 00:34
• Resolved: 03/Mar/15 00:34

Description

The fix for SPARK-2261 added a line at the beginning of the event log that encodes metadata. This line makes it much more difficult to parse the event logs from external libraries (like https://github.com/kayousterhout/trace-analysis, which is used by folks at Berkeley) because:

(1) The metadata is not written as JSON, unlike the rest of the file
(2) More annoyingly, if the file is compressed, the metadata is not compressed. This has a few side-effects: first, someone can't just use the command line to uncompress the file and then look at the logs, because the file is in this weird half-compressed format; and second, now external tools that parse these logs also need to deal with this weird format.

We should fix this before the 1.3 release, because otherwise we'll have to add a bunch more backward-compatibility code to handle this weird format!

Issue Links

• links to: [Github] Pull Request #4821 (andrewor14)