Details
-
Task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.0.0
-
None
Description
we need to change the nm retry settings to resolve the bug. The proposed new settings in yarn-site.xml are:
yarn.client.nodemanager-connect.max-wait-ms = 60,000
yarn.client.nodemanager-connect.retry-interval-ms = 10,000
If the underlying rpc retry takes 1 sec * 50 times to finish (50 secs), this will make each round of retry to be 10 + 50 secs = 1 minute. From the code, there will be 60,000 / 10,000 = 6 times of retries. So, in total, if the node manager accidentally crashed and never goes up, we well retry 1 minute * 6 times = 6 minutes. The whole calculation is based on the rpc settings (50 times, 1 sec sleep time),
Attachments
Attachments
Issue Links
- links to