It looks like Jenkins is no longer scheduling jobs. I attempted to restart the Jenkins agent via the UI on H4 (see below) and that's when things appears to have stopped getting scheduled.
Also, nodes H1, H2, H10, and H12 need to be kicked.
Additionally, I think I've discovered that the Hadoop HDFS unit tests for branch-2 are causing havoc on build nodes. ( tracking in
In any case, I'm still trying to reproduce the problem locally but it's tough going. I'm going to hard set which nodes certain tests run on to try and limit the damage though. Additionally, I've been working on
Anyway, sorry for the issues and thanks for the help.