Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12631

LLAP IO: support ORC ACID tables

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • llap, Transactions
    • None

    Description

      LLAP uses a completely separate read path in ORC to allow for caching and parallelization of reads and processing. This path does not support ACID. As far as I remember ACID logic is embedded inside ORC format; we need to refactor it to be on top of some interface, if practical; or just port it to LLAP read path.
      Another consideration is how the logic will work with cache. The cache is currently low-level (CB-level in ORC), so we could just use it to read bases and deltas (deltas should be cached with higher priority) and merge as usual. We could also cache merged representation in future.

      Attachments

        1. HIVE-12631.1.patch
          95 kB
          Teddy Choi
        2. HIVE-12631.2.patch
          102 kB
          Teddy Choi
        3. HIVE-12631.3.patch
          44 kB
          Teddy Choi
        4. HIVE-12631.4.patch
          44 kB
          Teddy Choi
        5. HIVE-12631.5.patch
          44 kB
          Teddy Choi
        6. HIVE-12631.6.patch
          47 kB
          Teddy Choi
        7. HIVE-12631.7.patch
          46 kB
          Teddy Choi
        8. HIVE-12631.8.patch
          46 kB
          Teddy Choi
        9. HIVE-12631.8.patch
          46 kB
          Teddy Choi
        10. HIVE-12631.9.patch
          71 kB
          Teddy Choi
        11. HIVE-12631.10.patch
          71 kB
          Teddy Choi
        12. HIVE-12631.10.patch
          73 kB
          Teddy Choi
        13. HIVE-12631.11.patch
          1.26 MB
          Teddy Choi
        14. HIVE-12631.11.patch
          112 kB
          Teddy Choi
        15. HIVE-12631.12.patch
          112 kB
          Teddy Choi
        16. HIVE-12631.13.patch
          166 kB
          Teddy Choi
        17. HIVE-12631.15.patch
          56 kB
          Teddy Choi
        18. HIVE-12631.16.patch
          63 kB
          Teddy Choi
        19. HIVE-12631.17.patch
          73 kB
          Teddy Choi
        20. HIVE-12631.18.patch
          161 kB
          Teddy Choi
        21. HIVE-12631.19.patch
          131 kB
          Teddy Choi
        22. HIVE-12631.20.patch
          73 kB
          Teddy Choi
        23. HIVE-12631.21.patch
          74 kB
          Teddy Choi
        24. HIVE-12631.22.patch
          76 kB
          Teddy Choi
        25. HIVE-12631.23.patch
          76 kB
          Teddy Choi
        26. HIVE-12631.24.patch
          97 kB
          Teddy Choi
        27. HIVE-12631.25.patch
          96 kB
          Teddy Choi
        28. HIVE-12631.26.patch
          97 kB
          Teddy Choi
        29. HIVE-12631.27.patch
          97 kB
          Sergey Shelukhin
        30. HIVE-12631.28.patch
          97 kB
          Sergey Shelukhin
        31. HIVE-12631.29.patch
          113 kB
          Teddy Choi
        32. HIVE-12631.30.patch
          114 kB
          Teddy Choi
        33. HIVE-12631.31.patch
          106 kB
          Teddy Choi

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. HIVE-17470 eliminate potential vector copies when merging ACID deltas in LLAP IO path

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-17458 VectorizedOrcAcidRowBatchReader doesn't handle 'original' files

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-17916 remove ConfVars.HIVE_VECTORIZATION_ROW_IDENTIFIER_ENABLED

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-12632 LLAP: don't use IO elevator for ACID tables

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-17854 LlapRecordReader should have getRowNumber() like org.apache.orc.RecordReader

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-15988 LLAP: cache ORC ACID delete deltas

          • Major - Major loss of function.
          • Resolved
          requires

          New Feature - A new feature of the product, which has yet to be developed. HIVE-14035 Enable predicate pushdown to delta files created by ACID Transactions

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link Review Board

          Activity

            People

              teddy.choi Teddy Choi
              sershe Sergey Shelukhin
              Votes:
              3 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: