Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22763

SHS: Ignore unknown events and parse through the file

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.1
    • Fix Version/s: 2.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      While spark code changes, there are new events in event log:
      https://github.com/apache/spark/pull/19649

      And we used to maintain a whitelist to avoid exceptions:
      https://github.com/apache/spark/pull/15663/files

      Currently Spark history server will stop parsing on unknown events or unrecognized properties. We may still see part of the UI data.

      For better compatibility, we can ignore unknown events and parse through the log file.

        Attachments

          Activity

            People

            • Assignee:
              Gengliang.Wang Gengliang Wang
              Reporter:
              Gengliang.Wang Gengliang Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: