[SPARK-26917] CacheManager blocks while traversing plans

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: SQL

Environment

We are running on AWS EMR 5.20, so Spark 2.4.0.

• Assignee: Dave DeCaprio
• Reporter: Dave DeCaprio
• Created: 18/Feb/19 20:34
• Updated: 25/Mar/19 03:54
• Resolved: 21/Feb/19 15:05

Description

This is related to SPARK-26548 and SPARK-26617.  The CacheManager is further locking during the recacheByCondition operation.  For large plans evaluation of the condition can take a long time.

Issue Links

• links to: GitHub Pull Request #23833
• links to: GitHub Pull Request #24028