[HUDI-5697] Spark SQL re-lists Hudi table after every SQL operations

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Fix Version/s: 0.14.0
• Component/s: spark, spark-sql
• Labels: pull-request-available
• Story Points: 4
• Epic Link: Performance Improvements
• Assignee: Alexey Kudinkin
• Reporter: Alexey Kudinkin
• Created: 03/Feb/23 21:41
• Updated: 1 week ago 05:31

Description

Currently, after most DML operations in Spark SQL, Hudi invokes `Catalog.refreshTable`

Prior to Spark 3.2, this was essentially doing the following:

1. Invalidating relation cache (forcing next time for relation to be re-resolved, creating new FileIndex, listing files, etc)
2. Trigger cascading invalidation (re-caching) of the cached data (in CacheManager)

As of Spark 3.2 it now additionally does `LogicalRelation.refresh` for ALL tables (previously this was only done for Temporary Views), therefore entailing whole table to be re-listed again by triggering `FileIndex.refresh` which might be costly operation.

 

We should revert back to preceding behavior from Spark 3.1

Issue Links

• links to: GitHub Pull Request #7847