[TEZ-3286] Allow clients to set processor reserved memory per vertex (instead of per container)

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.3
• Fix Version/s: 0.9.0, 0.8.4
• Assignee: Hitesh Shah
• Reporter: Wei Zheng
• Created: 02/Jun/16 23:43
• Updated: 24/Jul/16 03:30
• Resolved: 30/Jun/16 23:31

Description

tez.task.scale.memory.reserve-fraction can be set by clients to control how much memory is available to the processor. Ths values applies at a container level though, instead of at a vertex level.
In case of a hash-join - the processor typically needs more memory. In case of a Shuffle join - the processor may not need as much. In DAGs with a mix of map joins and shuffle joins - setting this at a container level is sub-optimal.

To a large extent this comes down to propagating vertex configs to the container / task.

Issue Links

• blocks: HIVE-13934 Configure Tez to make nocondiional task size memory available for the Processor
• is duplicated by: TEZ-2710 TEZ_TASK_SCALE_MEMORY_RESERVE_FRACTION should be configurable at vertex level
• is required by: PIG-4626 [Pig on Tez] OOM in case of multiple outputs and POPartialAgg