This test is racy and so the issue with it only manifests sometimes.
YARN-5773 put in a change so that verifyAMLimitForLeafQueue registers a 2GB node. queueA.getAMResourceLimit should return 1024 for memory, but is racy and almost always gets set to 0 for memory. So when the test submits the app with amResource2, that resource is set to 2048, instead of 3072. In the instances that it gets correctly set to 3072, the test errors out that it is trying to use more than the max memory available.
The bandaid fix would be to increase the node size to 4GB, but that doesn't fix the race in why queueA.getAMResourceLimit isn't getting updated before it is being accessed in the test.
Tan, Wangda, Varun Saxena, Bibin A Chundatt, Rohith Sharma K S, Naganarasimha G R, Sunil G: Does this analysis make sense? Should we just add a waitFor to wait until queueA.getAMResourceLimit isn't 0,0?