Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6270

Standalone Master hangs when streaming job completes and event logging is enabled

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.2.0, 1.2.1, 1.3.0, 1.5.1
    • Fix Version/s: None
    • Component/s: Deploy, DStreams
    • Labels:
      None

      Description

      If the event logging is enabled, the Spark Standalone Master tries to recreate the web UI of a completed Spark application from its event logs. However if this event log is huge (e.g. for a Spark Streaming application), then the master hangs in its attempt to read and recreate the web ui. This hang causes the whole standalone cluster to be unusable.

      Workaround is to disable the event logging.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                tdas Tathagata Das
              • Votes:
                4 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated: