Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20853

spark.ui.reverseProxy=true leads to hanging communication to master

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 2.1.0
    • Fix Version/s: None
    • Component/s: Web UI
    • Labels:
    • Environment:

      ppc64le GNU/Linux, POWER8, only master node is reachable externally other nodes are in an internal network

      Description

      When reverse proxy is enabled

      spark.ui.reverseProxy=true
      spark.ui.reverseProxyUrl=/

      first of all any invocation of the spark master Web UI hangs forever locally (e.g. http://192.168.10.16:25001) and via external URL without any data received.
      One, sometimes two spark applications succeed without error and than workers start throwing exceptions:

      Caused by: java.io.IOException: Failed to connect to /192.168.10.16:25050

      The application dies during creation of SparkContext:

      2017-05-22 16:11:23 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting to master spark://node0101:25000...
      2017-05-22 16:11:23 INFO TransportClientFactory:254 - Successfully created connection to node0101/192.168.10.16:25000 after 169 ms (132 ms spent in bootstraps)
      2017-05-22 16:11:43 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting to master spark://node0101:25000...
      2017-05-22 16:12:03 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting to master spark://node0101:25000...
      2017-05-22 16:12:23 ERROR StandaloneSchedulerBackend:70 - Application has been killed. Reason: All masters are unresponsive! Giving up.
      2017-05-22 16:12:23 WARN StandaloneSchedulerBackend:66 - Application ID is not initialized yet.
      2017-05-22 16:12:23 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 25056.
      .....
      Caused by: java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem

      This definitively does not happen without reverse proxy enabled!

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                byronstar Benno Staebler
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: