Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6270

Standalone Master hangs when streaming job completes and event logging is enabled

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Duplicate
    • 1.2.0, 1.2.1, 1.3.0, 1.5.1
    • None
    • Deploy, DStreams
    • None

    Description

      If the event logging is enabled, the Spark Standalone Master tries to recreate the web UI of a completed Spark application from its event logs. However if this event log is huge (e.g. for a Spark Streaming application), then the master hangs in its attempt to read and recreate the web ui. This hang causes the whole standalone cluster to be unusable.

      Workaround is to disable the event logging.

      Attachments

        Issue Links

          duplicates

          Sub-task - The sub-task of the issue SPARK-12299 Remove history serving functionality from standalone Master

          • Major - Major loss of function.
          • Resolved
          Is contained by

          Sub-task - The sub-task of the issue SPARK-12299 Remove history serving functionality from standalone Master

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-7600 Stopping Streaming Context (sometimes) crashes master

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-12108 Event logs are much bigger in 1.6 than in 1.5

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              tdas Tathagata Das
              Votes:
              4 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: