Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.7.0
-
None
-
None
-
None
Description
If there's a single bad NodeManager, and a lot of containers allocated on this node - all container launcher threads can end up blocked on this node, delaying subsequent launches.
This is despite timeouts kicking in.
1) We should not allow all threads to be used up for a single NM
2) The retry policy could be enhanced to stop at ConnectionTimeouts (e.g. Node down)
3) Interrupt launch requests once Tez has detected a container as timed out.
Noticed by rajesh.balamohan - threads would lockup for 15 minutes in 0.7, and potentially infinitely on 0.8. That's another bug that needs investigation in 0.8.