Hive
  1. Hive
  2. HIVE-5951

improve performance of adding partitions from client

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Adding partitions to metastore is currently very inefficient. There are small things like, for !ifNotExists case, DDLSemanticAnalyzer gets the full partition object for every spec (which is a network call to metastore), and then discards it instantly; there's also general problem that too much processing is done on client side. DDLSA should analyze the query and make one call to metastore (or maybe a set of batched calls if there are too many partitions in the command), metastore should then figure out stuff and insert in batch.

      1. HIVE-5951.07.patch
        718 kB
        Sergey Shelukhin
      2. HIVE-5951.07.patch
        718 kB
        Sergey Shelukhin
      3. HIVE-5951.05.patch
        715 kB
        Sergey Shelukhin
      4. HIVE-5951.04.patch
        713 kB
        Sergey Shelukhin
      5. HIVE-5951.03.patch
        707 kB
        Sergey Shelukhin
      6. HIVE-5951.nogen.patch
        89 kB
        Sergey Shelukhin
      7. HIVE-5951.02.patch
        707 kB
        Sergey Shelukhin
      8. HIVE-5951.nogen.patch
        89 kB
        Sergey Shelukhin
      9. HIVE-5951.01.patch
        705 kB
        Sergey Shelukhin
      10. HIVE-5951.nogen.patch
        87 kB
        Sergey Shelukhin
      11. HIVE-5951.patch
        703 kB
        Sergey Shelukhin
      12. HIVE-5951.nogen.patch
        85 kB
        Sergey Shelukhin

        Activity

        Sergey Shelukhin created issue -
        Sergey Shelukhin made changes -
        Field Original Value New Value
        Attachment HIVE-5951.nogen.patch [ 12617304 ]
        Attachment HIVE-5951.patch [ 12617305 ]
        Sergey Shelukhin made changes -
        Attachment HIVE-5951.nogen.patch [ 12617473 ]
        Attachment HIVE-5951.01.patch [ 12617474 ]
        Sergey Shelukhin made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Sergey Shelukhin made changes -
        Attachment HIVE-5951.nogen.patch [ 12617524 ]
        Sergey Shelukhin made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Sergey Shelukhin made changes -
        Attachment HIVE-5951.02.patch [ 12617525 ]
        Sergey Shelukhin made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Sergey Shelukhin made changes -
        Attachment HIVE-5951.nogen.patch [ 12617879 ]
        Attachment HIVE-5951.03.patch [ 12617880 ]
        Sergey Shelukhin made changes -
        Attachment HIVE-5951.04.patch [ 12622418 ]
        Sergey Shelukhin made changes -
        Attachment HIVE-5951.05.patch [ 12622457 ]
        Sergey Shelukhin made changes -
        Attachment HIVE-5951.07.patch [ 12622722 ]
        Sergey Shelukhin made changes -
        Attachment HIVE-5951.07.patch [ 12622970 ]
        Ashutosh Chauhan made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.13.0 [ 12324986 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Sergey Shelukhin
            Reporter:
            Sergey Shelukhin
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development