[SPARK-49149] Support customized log url for Spark UI and History server in Kubernetes environment

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: UI
• Assignee: Unassigned
• Reporter: Yichuan Huang
• Created: 07/Aug/24 23:13
• Updated: 15/Aug/24 07:16

Description

Spark provides two configs to alternate the log url on the live UI and history server, `spark.ui.custom.executor.log.url` and `spark.history.custom.executor.log.url`. The configs support path variables which will be replaced at runtime, but currently this only works on Yarn. Running on k8s with the path variable doesn't work. Here's an example

./bin/spark-shell --conf spark.ui.custom.executor.log.url="URL_PREFIX?appId=APP_ID&execId=EXECUTOR_ID"

The log column doesn't show up in the Spark UI, and Spark driver printing

24/08/07 17:23:45 INFO ExecutorLogUrlHandler: Fail to renew executor log urls: some of required attributes are missing in app's event log.. Required: Set(APP_ID, EXECUTOR_ID) / available: Set(). Falling back to s how app's original log urls.

 the empty attribute `Set()` in the above log segment indicates the feature not supported in k8s environment, thus creating this ticket to add the support