[GOBBLIN-1528] Reduce overshoot of Yarn containers in Gobblin-on-Yarn mode

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.15.0
• Fix Version/s: 0.17.0
• Component/s: gobblin-yarn
• Assignee: Sudarshan Vasudevan
• Reporter: Sudarshan Vasudevan
• Created: 27/Aug/21 22:35
• Updated: 14/Jun/23 04:13

Description

Gobblin can overallocate containers when a Yarn application is started up. The main reason for the overallocation is because the AMRM client does not track which container requests have already been submitted. This can result in container requests being re-sent and the consequent over-allocation. (See: YARN-1902). For Gobblin-on-Yarn, the overallocation results in creation of excess znodes by Helix, since each container maps to a Helix instance.

This task attempts to limit the container over-shoot during application start up, by explicitly tracking the number of inflight container requests.