Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11981

ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • None
    • 2.0.0
    • Hive, Transactions

    Description

      High priority issues with schema evolution for the ORC file format.

      Schema evolution here is limited to adding new columns and a few cases of column type-widening (e.g. int to bigint).

      Renaming columns, deleting column, moving columns and other schema evolution were not pursued due to lack of importance and lack of time. Also, it appears a much more sophisticated metadata would be needed to support them.

      The biggest issues for users have been adding new columns for ACID table (HIVE-11421 Support Schema evolution for ACID tables) and vectorization (HIVE-10598 Vectorization borks when column is added to table).

      Attachments

        1. ORC Schema Evolution Issues.docx
          112 kB
          Matt McCline
        2. HIVE-11981.01.patch
          367 kB
          Matt McCline
        3. HIVE-11981.02.patch
          396 kB
          Matt McCline
        4. HIVE-11981.03.patch
          432 kB
          Matt McCline
        5. HIVE-11981.05.patch
          602 kB
          Matt McCline
        6. HIVE-11981.06.patch
          626 kB
          Matt McCline
        7. HIVE-11981.07.patch
          626 kB
          Matt McCline
        8. HIVE-11981.08.patch
          628 kB
          Matt McCline
        9. HIVE-11981.09.patch
          631 kB
          Matt McCline
        10. HIVE-11981.091.patch
          598 kB
          Matt McCline
        11. HIVE-11981.092.patch
          608 kB
          Matt McCline
        12. HIVE-11981.093.patch
          628 kB
          Matt McCline
        13. HIVE-11981.094.patch
          662 kB
          Matt McCline
        14. HIVE-11981.095.patch
          666 kB
          Matt McCline
        15. HIVE-11981.096.patch
          1.18 MB
          Matt McCline
        16. HIVE-11981.097.patch
          1.11 MB
          Matt McCline
        17. HIVE-11981.098.patch
          1.12 MB
          Matt McCline
        18. HIVE-11981.099.patch
          1.12 MB
          Matt McCline
        19. HIVE-11981.0991.patch
          1.12 MB
          Matt McCline
        20. HIVE-11981.0992.patch
          1.12 MB
          Matt McCline

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. HIVE-11421 Support Schema evolution for ACID tables

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-12625 Backport to branch-1 HIVE-11981 ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-24472 Orc RecordReaderFactory throws IndexOutOfBoundsException

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-12728 Apply DDL restrictions for ORC schema evolution

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HIVE-12799 Always use Schema Evolution for ACID

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          links to

          Web Link RB #38820

          Activity

            People

              mmccline Matt McCline
              mmccline Matt McCline
              Votes:
              0 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: