[YARN-7086] Release all containers aynchronously

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: resourcemanager
• Target Version/s: 3.4.0
• Assignee: Manikandan R
• Reporter: Arun Suresh
• Created: 23/Aug/17 18:27
• Updated: 09/Apr/20 18:41

Description

We have noticed in production two situations that can cause deadlocks and cause scheduling of new containers to come to a halt, especially with regard to applications that have a lot of live containers:

1. When these applicaitons release these containers in bulk.
2. When these applications terminate abruptly due to some failure, the scheduler releases all its live containers in a loop.

To handle the issues mentioned above, we have a patch in production to make sure ALL container releases happen asynchronously - and it has served us well.

Opening this JIRA to gather feedback on if this is a good idea generally (cc Wangda Tan, Jason Darrell Lowe, Carlo Curino, Karthik Kambatla, Subramaniam Krishnan, Roni Burd)

BTW, In YARN-6251, we already have an asyncReleaseContainer() in the AbstractYarnScheduler and a corresponding scheduler event, which is currently used specifically for the container-update code paths (where the scheduler realeases temp containers which it creates for the update)