Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33507

Improve and fix cache behavior in v1 and v2

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • 3.1.0
    • None
    • SQL
    • None

    Description

      This is an umbrella JIRA to track fixes & improvements for caching behavior in Spark datasource v1 and v2, which includes:

      • fix existing cache behavior in v1 and v2.
      • fix inconsistent cache behavior between v1 and v2
      • implement missing features in v2 to align with those in v1.

      Attachments

        Issue Links

        1.
        DSv2: Append/Overwrite/ReplaceTable should invalidate cache Sub-task Resolved Chao Sun Actions
        2.
        DSv2: REFRESH TABLE should invalidate caches Sub-task Resolved Chao Sun Actions
        3.
        DSv2: DROP TABLE command should also invalidate cache Sub-task Resolved Chao Sun Actions
        4.
        REFRESH TABLE should invalidate cache even though the table itself may not be cached Sub-task Resolved Chao Sun Actions
        5.
        DSv2: DeleteFrom should refresh cache Sub-task Resolved Chao Sun Actions
        6.
        DSv2: REFRESH TABLE should recache the table itself Sub-task Resolved Chao Sun Actions
        7.
        [DSv2] ReplaceTable should invalidate cache Sub-task Resolved Chao Sun Actions
        8.
        When refreshing cache, Spark should not use cached plan when recaching data Sub-task Resolved Chao Sun Actions
        9.
        ALTER TABLE ... RENAME TO should recreate cache for v2 tables. Sub-task Resolved Terry Kim Actions
        10.
        A cached view should become invalid after a table is dropped Sub-task Resolved Chao Sun Actions
        11.
        ALTER TABLE .. DROP PARTITION uncaches Hive table while updating table stats Sub-task Resolved Max Gekk Actions
        12.
        Call updateTableStats() from AlterTableAddPartitionCommand Sub-task Resolved Unassigned Actions
        13.
        ALTER TABLE .. ADD PARTITION doesn't refresh cache Sub-task Resolved Max Gekk Actions
        14.
        ALTER TABLE .. RECOVER PARTITIONS doesn't refresh cache Sub-task Resolved Max Gekk Actions
        15.
        ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache Sub-task Resolved Max Gekk Actions
        16.
        `isCached` return `false` for cached Hive table Sub-task Resolved Max Gekk Actions
        17.
        ALTER TABLE .. DROP PARTITION doesn't refresh cache Sub-task Resolved Max Gekk Actions
        18.
        ALTER TABLE .. DROP PARTITION doesn't invalidate the cache Sub-task In Progress Unassigned Actions
        19.
        Improve semantics for REFRESH TABLE Sub-task Open Unassigned Actions
        20.
        Cache lookup doesn't work in certain cases Sub-task Resolved Unassigned Actions
        21.
        Keep dependants cached while refreshing v1 tables Sub-task Resolved Max Gekk Actions
        22.
        DSv2: `ALTER TABLE .. ADD PARTITION` does not refresh table cache Sub-task Resolved Max Gekk Actions
        23.
        LOAD DATA doesn't refresh v1 table cache Sub-task Resolved Max Gekk Actions
        24.
        Keep table cached after truncation Sub-task Resolved Max Gekk Actions
        25.
        ALTER TABLE .. SET LOCATION doesn't refresh v1 table cache Sub-task Resolved Max Gekk Actions
        26.
        Update comments for `SessionCatalog.refreshTable()` and `CatalogImpl.refreshTable()` Sub-task Resolved Max Gekk Actions
        27.
        Remove `refreshTable()` from `SessionState` Sub-task Resolved Max Gekk Actions
        28.
        CatalogImpl.uncacheTable should invalidate in cascade for temp views Sub-task Resolved Chao Sun Actions
        29.
        Update public docs about altering cached tables/views Sub-task Resolved Max Gekk Actions
        30.
        Streaming write to a V2 table should invalidate its associated cache Sub-task Resolved Chao Sun Actions

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            csun Chao Sun

            Dates

              Created:
              Updated:

              Slack

                Issue deployment