[SPARK-24281] Option to disable Spark UI's web filter in YARN-client mode

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.2.1, 2.3.0
• Component/s: Spark Core, Web UI, YARN
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Zoltán Zvara
• Created: 15/May/18 14:04
• Updated: 17/May/20 18:13
• Resolved: 08/Oct/19 05:41

Description

When Spark is deployed to YARN (either in client or cluster mode), the driver's web UI is accessible solely through YARN's web proxy due to security reasons. The YARN web proxy usually started on one of the {{ResourceManager}}s (RM) of YARN. A common use case for application developers is to deploy applications on the cluster directly from their computer in client mode. This ensures fast experiment deployments to the target cluster, while allows IDE-driven breakpoints on the driver, which is kept on the user's computer; in addition to keep all features that an IDE provides. In these setups, the user might be connected through a VPN or networks where packets that initiate connection might not find a route back to the driver. One example is when the user attempts to access the web UI running on their computer. The web UI will redirect to the proxy, which then attempts to access the web UI again. In this process, the proxy might fail to serve pages from the web UI: no response or redirect-loops can occur.

This improvement allows users in YARN-client mode to access the driver's web UI directly, bypassing the YARN proxy altogether.

In the event of the ApplicationMaster is started, a web UI filter will be applied on the endpoint, which then makes the web UI to redirect all calls not addressed from any of the registered RMs.

An option to disable the web UI filters would help users in the aforementioned scenarios to access the web UI from their local computer. An alternative is to keep their pre-registered filters even if the YARN web UI filters are applied - which now simply overwrites all user-specified filters.

A simple code to mitigate this problem and to make the web UI accessible locally in a YARN-client scenario is the following:

 

context.ui.foreach {
  _.getHandlers.asInstanceOf[Seq[ServletContextHandler]].foreach {
      h =>
        val filters = h.getServletHandler.getFilters
        if (filters.nonEmpty) {
          filters.foreach {
            _.doStop()
          }
          h.getServletHandler.setFilters(null)
          h.getServletHandler.setFilterMappings(null)
          System.setProperty("spark.ui.proxyBase", "")
        }
    }
}