Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-18839

Implement incremental rebuild for materialized views (only insert operations in source tables)

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Materialized views

    Description

      Implementation will follow current code path for full rebuild.

      When the MV query plan is retrieved, if the MV contents are outdated because there were insert operations in the source tables, we will introduce a filter with a condition based on stored value of ValidWriteIdLists. For instance, WRITE_ID < high_txn_id AND WRITE_ID NOT IN (x, y, ...). Then the rewriting will do the rest of the work by creating a partial rewriting, where the contents of the MV are read as well as the new contents from the source tables.

      This mechanism will not work only for ALTER MV... REBUILD, but also for user queries which will be able to benefit from using outdated MVs to compute part of the needed results.

      Attachments

        1. HIVE-18839.01.patch
          508 kB
          jcamachorodriguez
        2. HIVE-18839.02.patch
          1.15 MB
          jcamachorodriguez
        3. HIVE-18839.03.patch
          1.17 MB
          jcamachorodriguez
        4. HIVE-18839.04.patch
          1.17 MB
          jcamachorodriguez
        5. HIVE-18839.patch
          507 kB
          jcamachorodriguez

        Issue Links

          Activity

            People

              jcamacho Jesús Camacho Rodríguez
              jcamacho Jesús Camacho Rodríguez
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: