[SPARK-45616] Usages of ParVector are unsafe because it does not propagate ThreadLocals or SparkSession

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0, 3.5.1
• Component/s: Spark Core, SQL, Tests
• Labels: pull-request-available
• Assignee: Ankur Dave
• Reporter: Ankur Dave
• Created: 20/Oct/23 17:41
• Updated: 23/Oct/23 02:48
• Resolved: 23/Oct/23 02:48

Description

CastSuiteBase and ExpressionInfoSuite use ParVector.foreach() to run Spark SQL queries in parallel. They incorrectly assume that each parallel operation will inherit the main thread’s active SparkSession. This is only true when these parallel operations run in freshly-created threads. However, when other code has already run some parallel operations before Spark was started, then there may be existing threads that do not have an active SparkSession. In that case, these tests fail with NullPointerExceptions when creating SparkPlans or running SQL queries.

The fix is to use the existing method ThreadUtils.parmap(). This method creates fresh threads that inherit the current active SparkSession, and it propagates the Spark ThreadLocals.

We should also add a scalastyle warning against use of ParVector.

Issue Links

• links to: GitHub Pull Request #43466