Hive
  1. Hive
  2. HIVE-2026

Parallelize UpdateInputAccessTimeHook

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      UpdateInputAccessTimeHook is usually used as a pre-execution hook to update the metastore's lastAccessTime field of input partition/table. If a query touches a large number of partitions, this hooks takes a long time to execute. One approach is to make the hook itself to run in a separate thread. But it is hard to guarantee backward compatibility in semantics in case of exceptions encountered in the hook execution. This task takes another approach to parallelize the hook itself (update multiple partitions concurrently), but execute each pre-hook in sequential order.

      1. HIVE-2026_2.patch
        22 kB
        MIS
      2. HIVE-2026.patch
        22 kB
        Ning Zhang

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        1h 3m 1 Ning Zhang 03/Mar/11 08:54
        Patch Available Patch Available Open Open
        18d 11h 54m 1 Namit Jain 21/Mar/11 20:48
        Namit Jain made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Namit Jain added a comment -

        Ning, do you want the new parameter to use the new configuration variable - I mean, number of threads.
        I mean, ExecDriver can invoke the hooks in parallel and then wait for them, that way if we have
        a new hook with similar requirements in the future, we dont have ti duplicate this code.

        Show
        Namit Jain added a comment - Ning, do you want the new parameter to use the new configuration variable - I mean, number of threads. I mean, ExecDriver can invoke the hooks in parallel and then wait for them, that way if we have a new hook with similar requirements in the future, we dont have ti duplicate this code.
        Hide
        Ning Zhang added a comment -

        The review board has also been updated with the new HIVE-2026_2.patch

        Show
        Ning Zhang added a comment - The review board has also been updated with the new HIVE-2026 _2.patch
        MIS made changes -
        Attachment HIVE-2026_2.patch [ 12472651 ]
        Hide
        MIS added a comment -

        Patch incorporating the review comments.

        Show
        MIS added a comment - Patch incorporating the review comments.
        Ning Zhang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Ning Zhang added a comment -
        Show
        Ning Zhang added a comment - review board: https://reviews.apache.org/r/460/
        Ning Zhang made changes -
        Field Original Value New Value
        Attachment HIVE-2026.patch [ 12472515 ]
        Hide
        Ning Zhang added a comment -

        running tests and will update a review board.

        Show
        Ning Zhang added a comment - running tests and will update a review board.
        Ning Zhang created issue -

          People

          • Assignee:
            Ning Zhang
            Reporter:
            Ning Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development