[SPARK-16859] History Server storage information is missing

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.6.2, 2.0.0
• Component/s: Spark Core
• Labels: historyserver newbie
• Assignee: Unassigned
• Reporter: Andrei Ivanov
• Created: 02/Aug/16 18:23
• Updated: 09/Oct/20 18:44
• Resolved: 09/Oct/20 18:44

Description

It looks like job history storage tab in history server is broken for completed jobs since 1.6.2.
More specifically it's broken since SPARK-13845.

I've fixed for my installation by effectively reverting the above patch (see).

IMHO, the most straightforward fix would be to implement SparkListenerBlockUpdated serialization to JSON in JsonProtocol making sure it works from ReplayListenerBus.

The downside will be that it will still work incorrectly with pre patch job histories. But then, it doesn't work since 1.6.2 anyhow.

PS: I'd really love to have this fixed eventually. But I'm pretty new to Apache Spark and missing hands on Scala experience. So I'd prefer that it be fixed by someone experienced with roadmap vision. If nobody volunteers I'll try to patch myself.

Issue Links

• is caused by: SPARK-13845 BlockStatus and StreamBlockId keep on growing result driver OOM
• relates to: SPARK-27486 Enable History server storage information test
• links to: PR