Hive
  1. Hive
  2. HIVE-5095

Hive needs new operator walker for parallelization/optimization for tez

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: tez-branch
    • Fix Version/s: tez-branch
    • Component/s: Tez
    • Labels:
      None

      Description

      For tez to compute the number of reducers, we should be walking the operator tree in a topological fashion so that the reducers down the tree get the estimate from all parents. However, the current walkers in hive only walk the operator tree in a depth-first fashion. We need to add a new walker for the topological walk. Also, since information about the parent operators needs to be propagated on a per parent basis, we need to retain some context across operators to be passed to the child which the walker will co-ordinate.

      NO PRECOMMIT TESTS (this is wip for the tez branch)

      1. HIVE-5095.1.patch
        34 kB
        Gunther Hagleitner
      2. HIVE-5095.WIP.patch.txt
        47 kB
        Vikram Dixit K

        Issue Links

          Activity

          Hide
          Gunther Hagleitner added a comment -

          Committed .1 to branch. Thanks!

          Show
          Gunther Hagleitner added a comment - Committed .1 to branch. Thanks!
          Hide
          Vikram Dixit K added a comment -
          Show
          Vikram Dixit K added a comment - RB request. https://reviews.apache.org/r/13787/
          Hide
          Vikram Dixit K added a comment -

          Patch for the new walker.

          Show
          Vikram Dixit K added a comment - Patch for the new walker.

            People

            • Assignee:
              Vikram Dixit K
              Reporter:
              Vikram Dixit K
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development