Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6951

History server slow startup if the event log directory is large

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 2.4.0
    • Component/s: Web UI
    • Labels:
      None

      Description

      I started my history server, then navigated to the web UI where I expected to be able to view some completed applications, but the webpage was not available. It turned out that the History Server was not finished parsing all of the event logs in the event log directory that I had specified. I had accumulated a lot of event logs from months of running Spark, so it would have taken a very long time for the History Server to crunch through them all. I purged the event log directory and started from scratch, and the UI loaded immediately.

      We should have a pagination strategy or parse the directory lazily to avoid needing to wait after starting the history server.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vanzin Marcelo Masiero Vanzin
                Reporter:
                mcheah Matt Cheah
              • Votes:
                3 Vote for this issue
                Watchers:
                24 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: