Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
2.1.0, 2.2.0, 2.3.0, 2.4.0, 3.0.0
Description
Purpose: allow to run the Spark web UI behind a reverse proxy with URLs prefixed by a context root, like www.mydomain.com/spark. In particular, this allows to access multiple Spark clusters through the same virtual host, only distinguishing them by context root, like www.mydomain.com/cluster1, www.mydomain.com/cluster2, and it allows to run the Spark UI in a common cookie domain (for SSO) with other services.
SPARK-15487 introduced some support for front-end reverse proxies by allowing all Spark UI requests to be routed through the master UI as a single endpoint and also added a spark.ui.reverseProxyUrl setting to define a another proxy sitting in front of Spark. However, as noted in the comments on SPARK-15487, this mechanism does not currently work if the reverseProxyUrl includes a context root like the examples above: Most links generated by the Spark UI result in full path URLs (like /proxy/app-"id"/...) that do not account for a path prefix (context root) and work only if the Spark UI "owns" the entire virtual host. In fact, the only place in the UI where the reverseProxyUrl seems to be used is the back-link from the worker UI to the master UI.
The discussion on SPARK-15487 proposes to open a new issue for the problem, but that does not seem to have happened, so this issue aims to address the remaining shortcomings of spark.ui.reverseProxyUrl
The problem can be partially worked around by doing content rewrite in a front-end proxy and prefixing src="/..." or href="/..." links with a context root. However, detecting and patching URLs in HTML output is not a robust approach and breaks down for URLs included in custom REST responses. E.g. the "allexecutors" REST call used from the Spark 2.1.0 application/executors page returns links for log viewing that direct to the worker UI and do not work in this scenario.
This issue proposes to honor spark.ui.reverseProxyUrl throughout Spark UI URL generation. Experiments indicate that most of this can simply be achieved by using/prepending spark.ui.reverseProxyUrl to the existing spark.ui.proxyBase system property. Beyond that, the places that require adaption are
- worker and application links in the master web UI
- webui URLs returned by REST interfaces
Note: It seems that returned redirect location headers do not need to be adapted, since URL rewriting for these is commonly done in front-end proxies and has a well-defined interface
