Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11675

make use of file footer PPD API in ETL strategy or separate strategy

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.1.0
    • None

    Description

      Need to take a look at the best flow. It won't be much different if we do filtering metastore call for each partition. So perhaps we'd need the custom sync point/batching after all.
      Or we can make it opportunistic and not fetch any footers unless it can be pushed down to metastore or fetched from local cache, that way the only slow threaded op is directory listings

      Attachments

        1. HIVE-11675.premature.opti.patch
          16 kB
          Sergey Shelukhin
        2. HIVE-11675.patch
          45 kB
          Sergey Shelukhin
        3. HIVE-11675.14.patch
          107 kB
          Sergey Shelukhin
        4. HIVE-11675.13.patch
          107 kB
          Sergey Shelukhin
        5. HIVE-11675.12.patch
          107 kB
          Sergey Shelukhin
        6. HIVE-11675.11.patch
          105 kB
          Sergey Shelukhin
        7. HIVE-11675.10.patch
          2 kB
          Sergey Shelukhin
        8. HIVE-11675.09.patch
          109 kB
          Sergey Shelukhin
        9. HIVE-11675.08.patch
          109 kB
          Sergey Shelukhin
        10. HIVE-11675.07.patch
          109 kB
          Sergey Shelukhin
        11. HIVE-11675.06.patch
          87 kB
          Sergey Shelukhin
        12. HIVE-11675.05.patch
          85 kB
          Sergey Shelukhin
        13. HIVE-11675.04.patch
          45 kB
          Sergey Shelukhin
        14. HIVE-11675.03.patch
          45 kB
          Sergey Shelukhin
        15. HIVE-11675.02.patch
          44 kB
          Sergey Shelukhin
        16. HIVE-11675.01.patch
          46 kB
          Sergey Shelukhin

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. HIVE-12925 make sure metastore footer cache doesn't get all functions

          • Major - Major loss of function.
          • Open
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HIVE-11676 implement metastore API to do file footer PPD

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HIVE-11777 implement an option to have single ETL strategy for multiple directories

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HIVE-11856 allow split strategies to run on threadpool

          • Major - Major loss of function.
          • Closed
          is part of

          Sub-task - The sub-task of the issue HIVE-11500 implement file footer / splits cache in HBase metastore

          • Major - Major loss of function.
          • Open
          links to

          Web Link RB

          Activity

            People

              sershe Sergey Shelukhin
              sershe Sergey Shelukhin
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: