[SPARK-32039] Unable to Set `spark.ui.port` configuration in Yarn Cluster Mode

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 2.4.0
• Component/s: Web UI
• Assignee: Unassigned
• Reporter: rajat
• Created: 20/Jun/20 09:44
• Updated: 21/Jun/20 23:54
• Resolved: 21/Jun/20 23:54

Description

Spark Web UI port in Yarn cluster mode always gets a random number since we disable the configuration spark.ui.port by setting this property to be 0 always. 

There are some use cases where we need to run Web UI port on some specified range because of some security concerns and unable to do so with the configuration. 

 

https://github.com/apache/spark/blob/842b1dcdff0ecab4af9f292c2ff7b2b9ae1ac40a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L216

Issue Links

• duplicates: SPARK-29465 Unable to configure SPARK UI (spark.ui.port) in spark yarn cluster mode.
• links to: [Github] Pull Request #28879 (rajatahujaatinmobi)
• links to: [Github] Pull Request #28879 (rajatahujaatinmobi)
• links to: [Github] Pull Request #28880 (rajatahujaatinmobi)
• links to: [Github] Pull Request #28880 (rajatahujaatinmobi)