Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20582

Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Duplicate
    • 2.1.0
    • None
    • Deploy
    • None

    Description

      In the current code of HistoryServer,jetty server will be started after all the logs fetch from yarn has been replayed.However,when the number of logs is becoming larger,the start time of jetty will be too long.
      Here,we implement a solution which using ApplicationAttemptInfo checkpointing to speed up the start of historyserver.When historyserver is starting,it will load ApplicationAttemptInfo from checkpoint file first if exists which is faster then replaying one by one.

      Attachments

        Issue Links

          duplicates

          Umbrella - An overarching type made of sub-tasks SPARK-18085 SPIP: Better History Server scalability for many / large applications

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-21348 Spark history Server load too slow when eventlog is large

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              Unassigned Unassigned
              cane zhoukang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: