Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18085

SPIP: Better History Server scalability for many / large applications

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Umbrella
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.3.0
    • Component/s: Spark Core, Web UI
    • Labels:

      Description

      It's a known fact that the History Server currently has some annoying issues when serving lots of applications, and when serving large applications.

      I'm filing this umbrella to track work related to addressing those issues. I'll be attaching a document shortly describing the issues and suggesting a path to how to solve them.

        Attachments

        1. spark_hs_next_gen.pdf
          99 kB
          Marcelo Masiero Vanzin
        2. screenshot-1.png
          304 kB
          Jincheng Sun
        3. screenshot-2.png
          268 kB
          Jincheng Sun

        Issue Links

        1.
        Key-value store abstraction and implementation for storing application data Sub-task Resolved Marcelo Masiero Vanzin Actions
        2.
        Use key-value store to keep History Server application listing Sub-task Resolved Marcelo Masiero Vanzin Actions
        3.
        Implement listener for saving application status data in key-value store Sub-task Resolved Marcelo Masiero Vanzin Actions
        4.
        Hook up Spark UI to the new key-value store backend Sub-task Resolved Marcelo Masiero Vanzin Actions
        5.
        Make Environment page use new app state store Sub-task Resolved Marcelo Masiero Vanzin Actions
        6.
        Make Executors page use new app state store Sub-task Resolved Marcelo Masiero Vanzin Actions
        7.
        Make the Storage page use new app state store Sub-task Resolved Marcelo Masiero Vanzin Actions
        8.
        Make Jobs and Stages pages use the new app state store Sub-task Resolved Marcelo Masiero Vanzin Actions
        9.
        Simplify REST API class hierarchy Sub-task Resolved Marcelo Masiero Vanzin Actions
        10.
        Remove JobProgressListener (and other unneeded classes) Sub-task Resolved Marcelo Masiero Vanzin Actions
        11.
        Speed up the new app state listener Sub-task Resolved Unassigned Actions
        12.
        Make SQL UI use new app state store Sub-task Resolved Marcelo Masiero Vanzin Actions
        13.
        Add auto-cleanup of old elements to the new app state store Sub-task Resolved Marcelo Masiero Vanzin Actions
        14.
        Add controls for how much disk the SHS can use Sub-task Resolved Marcelo Masiero Vanzin Actions
        15.
        In-memory key-value store implementation Sub-task Resolved Marcelo Masiero Vanzin Actions
        16.
        Speed up Stage page Sub-task Resolved Marcelo Masiero Vanzin Actions
        17.
        Remove stale applications from SHS listing Sub-task Resolved Marcelo Masiero Vanzin Actions
        18.
        Move kvstore package to util.kvstore, add annotations Sub-task Resolved Marcelo Masiero Vanzin Actions

          Activity

            People

            • Assignee:
              vanzin Marcelo Masiero Vanzin
              Reporter:
              vanzin Marcelo Masiero Vanzin
              Shepherd:
              Marcelo Masiero Vanzin

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment