Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-860

Persistent distributed cache

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 0.12.0
    • None
    • None
    • None

    Description

      DistributedCache is shared across multiple jobs, if the hdfs file name is the same.

      We need to make sure Hive put the same file into the same location every time and do not overwrite if the file content is the same.

      We can achieve 2 different results:
      A1. Files added with the same name, timestamp, and md5 in the same session will have a single copy in distributed cache.
      A2. Filed added with the same name, timestamp, and md5 will have a single copy in distributed cache.

      A2 has a bigger benefit in sharing but may raise a question on when Hive should clean it up in hdfs.

      Attachments

        1. HIVE-860.1.patch
          20 kB
          Ferdinand Xu
        2. HIVE-860.2.patch
          20 kB
          Ferdinand Xu
        3. HIVE-860.2.patch
          20 kB
          Ferdinand Xu
        4. HIVE-860.3.patch
          20 kB
          Ferdinand Xu
        5. HIVE-860.4.patch
          21 kB
          Brock Noland
        6. HIVE-860.4.patch
          21 kB
          Dong Chen
        7. HIVE-860.4.patch
          21 kB
          Ferdinand Xu
        8. HIVE-860.4.patch
          21 kB
          Dong Chen
        9. HIVE-860.4.patch
          20 kB
          Ferdinand Xu
        10. HIVE-860.4.patch
          20 kB
          Ferdinand Xu
        11. HIVE-860.5.patch
          21 kB
          Brock Noland
        12. HIVE-860.patch
          22 kB
          Brock Noland
        13. HIVE-860.patch
          22 kB
          Brock Noland
        14. HIVE-860.patch
          21 kB
          Brock Noland
        15. HIVE-860.patch
          21 kB
          Brock Noland
        16. HIVE-860.patch
          21 kB
          Brock Noland
        17. HIVE-860.patch
          21 kB
          Brock Noland
        18. HIVE-860.patch
          19 kB
          Brock Noland
        19. HIVE-860.patch
          19 kB
          Brock Noland
        20. HIVE-860.patch
          18 kB
          Brock Noland
        21. HIVE-860.patch
          18 kB
          Brock Noland
        22. HIVE-860.patch
          17 kB
          Brock Noland
        23. HIVE-860-debug.4.patch
          21 kB
          Dong Chen

        Issue Links

          Activity

            People

              dongc Dong Chen
              zshao Zheng Shao
              Votes:
              2 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated: