[SPARK-30263] Don't log values of ignored non-Spark properties

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.4, 3.0.0
• Fix Version/s: 2.4.5, 3.0.0
• Component/s: Spark Core
• Assignee: Sean R. Owen
• Reporter: Sean R. Owen
• Created: 14/Dec/19 13:43
• Updated: 14/Dec/19 21:14
• Resolved: 14/Dec/19 21:14

Description

Comment per Aaron Steers:

Is it expected that this error would print aws security keys to log files? Seems like a serious security concern.

Warning: Ignoring non-spark config property: fs.s3a.access.key={full-access-key}
Warning: Ignoring non-spark config property: fs.s3a.secret.key={full-secret-key}

Could we not accomplish the same thing by printing the name of the key without the key's value?

I think we can also redact these, but, also no big reason to log the value of ignored properties here anyway.

Issue Links

• links to: GitHub Pull Request #26893