[TEZ-4213] Bound appContext executor capacity using a configurable property

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Labels: 0.10_blocker
• Assignee: Panagiotis Garefalakis
• Reporter: Panagiotis Garefalakis
• Created: 30/Jul/20 22:04
• Updated: 15/Dec/20 19:35
• Resolved: 24/Aug/20 12:45

Description

After TEZ-4170 was merged, appContext executor pool is also used by the RootInputInitializerManager to speed up SplitGeneration.

However, this executor pool currently has no capacity limit https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java#L624

The problem occurs when generating splits for larger inputs (thousands or more) is that it can could result to
java.lang.OutOfMemoryError
that is also reproducible with a test case.
https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/dag/RootInputInitializerManager.java#L130

To avoid such errors, I propose to limit the capacity of this pool to a configurable value that can be for example the number of physical cores by default.

Issue Links

• links to: Github PR