Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20582

Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 2.1.0
    • Fix Version/s: None
    • Component/s: Deploy
    • Labels:
      None

      Description

      In the current code of HistoryServer,jetty server will be started after all the logs fetch from yarn has been replayed.However,when the number of logs is becoming larger,the start time of jetty will be too long.
      Here,we implement a solution which using ApplicationAttemptInfo checkpointing to speed up the start of historyserver.When historyserver is starting,it will load ApplicationAttemptInfo from checkpoint file first if exists which is faster then replaying one by one.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                cane zhoukang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: