Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20656

Incremental parsing of event logs in SHS

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 2.3.0
    • None
    • Spark Core

    Description

      This feature is mentioned in the spec attached to SPARK-18085 but there's not a lot of discussion about it.

      It would be good to implement incremental parsing of event logs in the SHS. With the new work, UI data is stored on disk, so it should be possible to save enough metadata about the event log and the state of the listeners to allow one to resume parsing the log of a live application at the point where it stopped in the previous iteration.

      This would considerably speed up parsing on updates, and could be done speculatively so that UIs for new applications are available in the SHS almost immediately.

      I'm filing this as a separate enhancement because I don't want to block SPARK-18085 on this.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            vanzin Marcelo Masiero Vanzin
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment