[SPARK-48946] NPE in DataSourceV2ScanExecBase redact method when session is null

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Mikó Szilárd
• Reporter: Mikó Szilárd
• Created: 19/Jul/24 08:35
• Updated: 22/Jul/24 09:35
• Resolved: 19/Jul/24 14:59

Description

If we call DataSourceV2ScanExecBase redact method from a thread that don't have a session in thread local we get an NPE.

https://github.com/apache/spark/blob/3f6e2d6ef354c034ce0b39a092d23dd99431a16c/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2ScanExecBase.scala#L70C3-L75C4

 

Changing 

session.sessionState.conf.stringRedactionPattern 

to 

conf.stringRedactionPattern  

could prevent this problem as conf checks if session is null or not.

https://github.com/apache/spark/blob/af5eb085b0ac01eec2bb5db206b7d499d462abc4/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala#L69

In case of DataSourceScanExec we already use conf.stringRedactionPattern:
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L93-L95

Issue Links

• links to: GitHub Pull Request #47419