[SPARK-33504] The application log in the Spark history server contains sensitive attributes such as password that should be redated instead of plain text

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 3.0.3, 3.1.0
• Component/s: Spark Core

Environment

Spark 3.0.1

• Assignee: huangkunlun
• Reporter: huangkunlun
• Created: 20/Nov/20 15:54
• Updated: 24/Feb/21 08:48
• Resolved: 02/Dec/20 15:28

Description

We found the secure attributes in SparkListenerJobStart and SparkListenerStageSubmitted events would not been redated, resulting in sensitive attributes can be viewd directly.

Issue Links

• links to: [Github] Pull Request #30446 (akiyamaneko)
• links to: [Github] Pull Request #30576 (tgravescs)
• links to: [Github] Pull Request #30576 (tgravescs)
• links to: [Github] Pull Request #31631 (viirya)
• links to: [Github] Pull Request #31631 (viirya)