Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Duplicate
-
2.5.0, 2.5.2
-
None
-
None
Description
If port of "yarn.nodemanager.address" is not specified at NM, NM will choose random port. If the NM is ungracefully dead(OOM kill, kill -9, or OS restart) and then restarted within "yarn.nm.liveness-monitor.expiry-interval-ms", "host:port1" and "host:port2" will both be present in "Active Nodes" on WebUI for a while, and after host:port1 expiration, we get host:port1 in "Lost Nodes" and host:port2 in "Active Nodes". If the NM is ungracefully dead again, we get only host:port1 in "Lost Nodes". "host:port2" is neither in "Active Nodes" nor in "Lost Nodes".
Another case, two NM is running on same host(miniYarnCluster or other test purpose), if both of them are lost, we get only one "Lost Nodes" in WebUI.
In both case, sum of "Active Nodes" and "Lost Nodes" is not the number of nodes we expected.
The root cause is due to inconsistency at how we think two Nodes are identical.
When we manager active nodes(RMContextImpl.nodes), we use NodeId which contains port. Two nodes with same host but different port are thought to be different node.
But when we manager inactive nodes(RMContextImpl.inactiveNodes), we use only use host. Two nodes with same host but different port are thought to identical.
To fix the inconsistency, we should differentiate below 2 cases and be consistent for both of them:
- intentionally multiple NMs per host
- NM instances one after another on same host
Two possible solutions:
1) Introduce a boolean config like "one-node-per-host"(default as "true"), and use host to differentiate nodes on RM if it's true.
2) Make it mandatory to have valid port in "yarn.nodemanager.address" config. In this sutiation, NM instances one after another on same host will have same NodeId, while intentionally multiple NMs per host will have different NodeId.
Personally I prefer option 1 because it's easier for users.