[SPARK-35328] Use 'SPARK_LOG_URL_' as env prefix for getting driver log urls by default

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.1
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: sharkd tu
• Created: 06/May/21 14:03
• Updated: 06/May/21 14:11

Description

Currently, spark on kubernetes can't show logs url on ui. To check history logs, we usually collect pod logs to third-party logging services, which can be accessed by urls. To show log urls,  we can set env prefixed with 'SPARK_LOG_URL_' for executors. But for driver, there is no way to show log urls by set env. 

 

I will create a new pr that use 'SPARK_LOG_URL_' as env prefix for getting driver log urls by default.

Issue Links

• links to: [Github] Pull Request #32456 (sharkdtu)
• links to: [Github] Pull Request #32456 (sharkdtu)