Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5522

Accelerate the History Server start

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.4.0
    • Component/s: Spark Core, Web UI
    • Labels:
      None

      Description

      When starting the history server, all the log files will be fetched and parsed in order to get the applications' meta data e.g. App Name, Start Time, Duration, etc. In our production cluster, there exist 2600 log files (160G) in HDFS and it costs 3 hours to restart the history server, which is a little bit too long for us.

      It would be better, if the history server can show logs with missing information during start-up and fill the missing information after fetching and parsing a log file.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                marsishandsome Liangliang Gu
                Reporter:
                marsishandsome Liangliang Gu
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: