[SPARK-26548] Don't block during query optimization

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Labels: sql
• Assignee: Dave DeCaprio
• Reporter: Dave DeCaprio
• Created: 05/Jan/19 15:59
• Updated: 06/Jan/19 03:24
• Resolved: 06/Jan/19 03:22

Description

In Spark 2.4.0 the CacheManager was updated so it will not execute jobs while it holds a lock.This was introduced in SPARK-23880.

The CacheManager still holds a write lock during the execution of the query optimizer.  For complex queries the optimizer can run for a long time (we see 10-15 minutes for some exceptionally large queries).  This allows only 1 thread to optimize at once. 

Issue Links

• links to: GitHub Pull Request #23469