Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16859

History Server storage information is missing

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.6.2, 2.0.0
    • Fix Version/s: None
    • Component/s: Spark Core

      Description

      It looks like job history storage tab in history server is broken for completed jobs since 1.6.2.
      More specifically it's broken since SPARK-13845.

      I've fixed for my installation by effectively reverting the above patch (see).

      IMHO, the most straightforward fix would be to implement SparkListenerBlockUpdated serialization to JSON in JsonProtocol making sure it works from ReplayListenerBus.

      The downside will be that it will still work incorrectly with pre patch job histories. But then, it doesn't work since 1.6.2 anyhow.

      PS: I'd really love to have this fixed eventually. But I'm pretty new to Apache Spark and missing hands on Scala experience. So I'd prefer that it be fixed by someone experienced with roadmap vision. If nobody volunteers I'll try to patch myself.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Hauer Andrei Ivanov
              • Votes:
                3 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated: