Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17874

Additional SSL port on HistoryServer should be configurable

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.2.0
    • Component/s: Web UI
    • Labels:
      None

      Description

      When turning on SSL on the HistoryServer with spark.ssl.historyServer.enabled=true this opens up a second port, at the hardcoded result of calculating spark.history.ui.port + 400, and sets up a redirect from the original (http) port to the new (https) port.

      $ netstat -nlp | grep 23714
      (Not all processes could be identified, non-owned process info
       will not be shown, you would have to be root to see it all.)
      tcp        0      0 :::18080                    :::*                        LISTEN      23714/java
      tcp        0      0 :::18480                    :::*                        LISTEN      23714/java
      

      By enabling spark.ssl.historyServer.enabled I would have expected the one open port to change protocol from http to https, not to have 1) additional ports open 2) the http port remain open 3) the additional port at a value I didn't specify.

      To fix this could take one of two approaches:

      Approach 1:

      • one port always, which is configured with spark.history.ui.port
      • the protocol on that port is http by default
      • or if spark.ssl.historyServer.enabled=true then it's https

      Approach 2:

      • add a new configuration item spark.history.ui.sslPort which configures the second port that starts up

      In approach 1 we probably need a way to specify to Spark jobs whether the history server has ssl or not, based on SPARK-16988

      That makes me think we should go with approach 2.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vanzin Marcelo Masiero Vanzin
                Reporter:
                aash Andrew Ash
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: