As a user can make a new reservation, the front-end can assign a node that is currently being cleaned up from a deleted reservation. So far the states observed are:
currentstate = pending
laststate = deleted
node state = reserved
If this node is assigned to a new reservation, the back end checks for existing process, logs it and fails the new reservation. It fails it likely because the new process on the backend does not know how much time is left to clean up the deleted reservation.
A decision/implementation on synchronizing the front-end and back-end needs to be made on how to handle this case as to not have reservation failures.
Suggestions are to:
1) only select available machines not assigned to reservations on the front end.
2) Or on the backend - wait until the previous deleted process is complete( which could take a while depending on what can be done)
3) on the backend - intercept the process and force a reload - not matter what the image is.