Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-5951

improve performance of adding partitions from client

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Adding partitions to metastore is currently very inefficient. There are small things like, for !ifNotExists case, DDLSemanticAnalyzer gets the full partition object for every spec (which is a network call to metastore), and then discards it instantly; there's also general problem that too much processing is done on client side. DDLSA should analyze the query and make one call to metastore (or maybe a set of batched calls if there are too many partitions in the command), metastore should then figure out stuff and insert in batch.

        Attachments

        1. HIVE-5951.nogen.patch
          85 kB
          Sergey Shelukhin
        2. HIVE-5951.patch
          703 kB
          Sergey Shelukhin
        3. HIVE-5951.nogen.patch
          87 kB
          Sergey Shelukhin
        4. HIVE-5951.01.patch
          705 kB
          Sergey Shelukhin
        5. HIVE-5951.nogen.patch
          89 kB
          Sergey Shelukhin
        6. HIVE-5951.02.patch
          707 kB
          Sergey Shelukhin
        7. HIVE-5951.nogen.patch
          89 kB
          Sergey Shelukhin
        8. HIVE-5951.03.patch
          707 kB
          Sergey Shelukhin
        9. HIVE-5951.04.patch
          713 kB
          Sergey Shelukhin
        10. HIVE-5951.05.patch
          715 kB
          Sergey Shelukhin
        11. HIVE-5951.07.patch
          718 kB
          Sergey Shelukhin
        12. HIVE-5951.07.patch
          718 kB
          Sergey Shelukhin

          Activity

            People

            • Assignee:
              sershe Sergey Shelukhin
              Reporter:
              sershe Sergey Shelukhin
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: