[HIVE-5095] Hive needs new operator walker for parallelization/optimization for tez

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: tez-branch
• Fix Version/s: tez-branch
• Component/s: Tez
• Assignee: Vikram Dixit K
• Reporter: Vikram Dixit K
• Created: 14/Aug/13 22:30
• Updated: 29/Aug/13 23:33
• Resolved: 29/Aug/13 23:32

Description

For tez to compute the number of reducers, we should be walking the operator tree in a topological fashion so that the reducers down the tree get the estimate from all parents. However, the current walkers in hive only walk the operator tree in a depth-first fashion. We need to add a new walker for the topological walk. Also, since information about the parent operators needs to be propagated on a per parent basis, we need to retain some context across operators to be passed to the child which the walker will co-ordinate.

NO PRECOMMIT TESTS (this is wip for the tez branch)

Issue Links

• blocks: HIVE-4660 Let there be Tez
• duplicates: HIVE-5052 Set parallelism when generating the tez tasks