[SPARK-41365] Stages UI page fails to load for proxy in some yarn versions - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.3.1
Fix Version/s: 3.3.2, 3.4.0
Component/s: Web UI
Labels:
None
Environment:

as above

Description

My environment CDH 5.8 , click to enter the spark UI from the yarn interface
when visit the stage URI, it fails to load, URI is

http://<yarn-url>:8088/proxy/application_1669877165233_0021/stages/stage/?id=0&attempt=0

Server error stack trace:

Caused by: java.lang.NullPointerException
	at org.apache.spark.status.api.v1.StagesResource.$anonfun$doPagination$1(StagesResource.scala:207)
	at org.apache.spark.status.api.v1.BaseAppResource.$anonfun$withUI$1(ApiRootResource.scala:142)
	at org.apache.spark.ui.SparkUI.withSparkUI(SparkUI.scala:147)
	at org.apache.spark.status.api.v1.BaseAppResource.withUI(ApiRootResource.scala:137)
	at org.apache.spark.status.api.v1.BaseAppResource.withUI$(ApiRootResource.scala:135)
	at org.apache.spark.status.api.v1.StagesResource.withUI(StagesResource.scala:31)
	at org.apache.spark.status.api.v1.StagesResource.doPagination(StagesResource.scala:206)
	at org.apache.spark.status.api.v1.StagesResource.$anonfun$taskTable$1(StagesResource.scala:161)
	at org.apache.spark.status.api.v1.BaseAppResource.$anonfun$withUI$1(ApiRootResource.scala:142)
	at org.apache.spark.ui.SparkUI.withSparkUI(SparkUI.scala:147)
	at org.apache.spark.status.api.v1.BaseAppResource.withUI(ApiRootResource.scala:137)
	at org.apache.spark.status.api.v1.BaseAppResource.withUI$(ApiRootResource.scala:135)
	at org.apache.spark.status.api.v1.StagesResource.withUI(StagesResource.scala:31)
	at org.apache.spark.status.api.v1.StagesResource.taskTable(StagesResource.scala:145)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

The issue is similar to, the final phenomenon of the issue is the same, because the parameter encode twice
https://issues.apache.org/jira/browse/SPARK-32467
https://issues.apache.org/jira/browse/SPARK-33611

The two issues solve two scenarios to avoid encode twice:
1. https redirect proxy
2. set reverse proxy enabled (spark.ui.reverseProxy) in Nginx

But if encode twice due to other reasons, such as this issue (yarn proxy), it will also fail