Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21348

Spark history Server load too slow when eventlog is large

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 2.1.0
    • Fix Version/s: None
    • Component/s: Web UI
    • Labels:
      None

      Description

      There is a eventlog file in hdfs as below:
      ```
      rwxrwx-- 3 root supergroup 9634238709 2017-07-08 04:37 hdfs://xxxxx/shared/spark-logs/xxxxx/xxxx.lz4
      ```

      and i start a spark history server to loading that eventlog file,but it is too slow to get spark web ui through that spark history server.

      it has requested for several hours but nothing about spark application web ui appears in the browser

      And the cpu usage of spark history server is 105.3%
      total used-memory in host is 9G

      The host has 48 Cores+130G Memory,enough Computing resources for spark history server.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                duyanghao duyanghao
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: