[HIVE-7158] Use Tez auto-parallelism in Hive

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Assignee: Gunther Hagleitner
• Reporter: Gunther Hagleitner
• Created: 31/May/14 06:48
• Updated: 27/Jan/15 11:11
• Resolved: 13/Jun/14 05:12

Description

Tez can optionally sample data from a fraction of the tasks of a vertex and use that information to choose the number of downstream tasks for any given scatter gather edge.

Hive estimates the count of reducers by looking at stats and estimates for each operator in the operator pipeline leading up to the reducer. However, if this estimate turns out to be too large, Tez can reign in the resources used to compute the reducer.

It does so by combining partitions of the upstream vertex. It cannot, however, add reducers at this stage.

I'm proposing to let users specify whether they want to use auto-parallelism or not. If they do there will be scaling factors to determine max and min reducers Tez can choose from. We will then partition by max reducers, letting Tez sample and reign in the count up until the specified min.

Issue Links

• depends upon: HIVE-7121 Use murmur hash to distribute HiveKey
• depends upon: TEZ-1162 Tez leaks CodecPool buffers
• depends upon: TEZ-1163 Tez Auto Reducer-parallelism throws Divide-by-Zero
• relates to: HIVE-7917 Hive max reducers count has regressed from a prime number to 999 (re-apply HIVE-7158)
• relates to: HIVE-7277 how to decide reduce numbers according to the input size of reduce stage rather than the input size of map stage?