Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
2.4.0
Description
When performing non-cascading cache invalidation, recache is called on the other cache entries which are dependent on the cache being invalidated. It leads to the the physical plans of those cache entries being re-compiled. For those cache entries, if the cache RDD has already been persisted, chances are there will be inconsistency between the data and the new plan. It can cause a correctness issue if the new plan's outputPartitioning or outputOrdering is different from the that of the actual data, and meanwhile the cache is used by another query that asks for specific outputPartitioning or outputOrdering which happens to match the new plan but not the actual data.