Hive
  1. Hive
  2. HIVE-2026

Parallelize UpdateInputAccessTimeHook

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      UpdateInputAccessTimeHook is usually used as a pre-execution hook to update the metastore's lastAccessTime field of input partition/table. If a query touches a large number of partitions, this hooks takes a long time to execute. One approach is to make the hook itself to run in a separate thread. But it is hard to guarantee backward compatibility in semantics in case of exceptions encountered in the hook execution. This task takes another approach to parallelize the hook itself (update multiple partitions concurrently), but execute each pre-hook in sequential order.

      1. HIVE-2026.patch
        22 kB
        Ning Zhang
      2. HIVE-2026_2.patch
        22 kB
        MIS

        Activity

        Hide
        Namit Jain added a comment -

        Ning, do you want the new parameter to use the new configuration variable - I mean, number of threads.
        I mean, ExecDriver can invoke the hooks in parallel and then wait for them, that way if we have
        a new hook with similar requirements in the future, we dont have ti duplicate this code.

        Show
        Namit Jain added a comment - Ning, do you want the new parameter to use the new configuration variable - I mean, number of threads. I mean, ExecDriver can invoke the hooks in parallel and then wait for them, that way if we have a new hook with similar requirements in the future, we dont have ti duplicate this code.
        Hide
        Ning Zhang added a comment -

        The review board has also been updated with the new HIVE-2026_2.patch

        Show
        Ning Zhang added a comment - The review board has also been updated with the new HIVE-2026 _2.patch
        Hide
        MIS added a comment -

        Patch incorporating the review comments.

        Show
        MIS added a comment - Patch incorporating the review comments.
        Hide
        Ning Zhang added a comment -
        Show
        Ning Zhang added a comment - review board: https://reviews.apache.org/r/460/
        Hide
        Ning Zhang added a comment -

        running tests and will update a review board.

        Show
        Ning Zhang added a comment - running tests and will update a review board.

          People

          • Assignee:
            Ning Zhang
            Reporter:
            Ning Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development