Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33507

Improve and fix cache behavior in v1 and v2

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • 3.1.0
    • None
    • SQL
    • None

    Description

      This is an umbrella JIRA to track fixes & improvements for caching behavior in Spark datasource v1 and v2, which includes:

      • fix existing cache behavior in v1 and v2.
      • fix inconsistent cache behavior between v1 and v2
      • implement missing features in v2 to align with those in v1.

      Attachments

        Issue Links

          relates to

          Umbrella - An overarching type made of sub-tasks SPARK-33392 Align DSv2 commands to DSv1 implementation

          • Major - Major loss of function.
          • Open
          1.
          		 DSv2: Append/Overwrite/ReplaceTable should invalidate cache Sub-task Resolved Chao Sun
          2.
          		 DSv2: REFRESH TABLE should invalidate caches Sub-task Resolved Chao Sun
          3.
          		 DSv2: DROP TABLE command should also invalidate cache Sub-task Resolved Chao Sun
          4.
          		 REFRESH TABLE should invalidate cache even though the table itself may not be cached Sub-task Resolved Chao Sun
          5.
          		 DSv2: DeleteFrom should refresh cache Sub-task Resolved Chao Sun
          6.
          		 DSv2: REFRESH TABLE should recache the table itself Sub-task Resolved Chao Sun
          7.
          		 [DSv2] ReplaceTable should invalidate cache Sub-task Resolved Chao Sun
          8.
          		 When refreshing cache, Spark should not use cached plan when recaching data Sub-task Resolved Chao Sun
          9.
          		 ALTER TABLE ... RENAME TO should recreate cache for v2 tables. Sub-task Resolved Terry Kim
          10.
          		 A cached view should become invalid after a table is dropped Sub-task Resolved Chao Sun
          11.
          		 ALTER TABLE .. DROP PARTITION uncaches Hive table while updating table stats Sub-task Resolved Max Gekk
          12.
          		 Call updateTableStats() from AlterTableAddPartitionCommand Sub-task Resolved Unassigned
          13.
          		 ALTER TABLE .. ADD PARTITION doesn't refresh cache Sub-task Resolved Max Gekk
          14.
          		 ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache Sub-task Resolved Max Gekk
          15.
          		 ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache Sub-task Resolved Max Gekk
          16.
          		 `isCached` return `false` for cached Hive table Sub-task Resolved Max Gekk
          17.
          		 ALTER TABLE .. DROP PARTITION doesn't refresh cache Sub-task Resolved Max Gekk
          18.
          		 ALTER TABLE .. DROP PARTITION doesn't invalidate the cache Sub-task In Progress Unassigned
          19.
          		 Improve semantics for REFRESH TABLE Sub-task Open Unassigned
          20.
          		 Cache lookup doesn't work in certain cases Sub-task Resolved Unassigned
          21.
          		 Keep dependants cached while refreshing v1 tables Sub-task Resolved Max Gekk
          22.
          		 DSv2: `ALTER TABLE .. ADD PARTITION` does not refresh table cache Sub-task Resolved Max Gekk
          23.
          		 LOAD DATA doesn't refresh v1 table cache Sub-task Resolved Max Gekk
          24.
          		 Keep table cached after truncation Sub-task Resolved Max Gekk
          25.
          		 ALTER TABLE .. SET LOCATION doesn't refresh v1 table cache Sub-task Resolved Max Gekk
          26.
          		 Update comments for `SessionCatalog.refreshTable()` and `CatalogImpl.refreshTable()` Sub-task Resolved Max Gekk
          27.
          		 Remove `refreshTable()` from `SessionState` Sub-task Resolved Max Gekk
          28.
          		 CatalogImpl.uncacheTable should invalidate in cascade for temp views Sub-task Resolved Chao Sun
          29.
          		 Update public docs about altering cached tables/views Sub-task Resolved Max Gekk
          30.
          		 Streaming write to a V2 table should invalidate its associated cache Sub-task Resolved Chao Sun

          Activity

            People

              Unassigned Unassigned
              csun Chao Sun
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: