Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.0.0-Ducc
-
None
Description
Its been observed that an Agent killed a JD immediately after starting it. This happens rarely but does occur.Looking through the Agent log I see that multiple OR state updates arrive almost instantly. In one of the states the OR requests a new process to be started while in the second state the process doesnt even exist. So the agent first launches the process and then subsequently kills it as this indicates a ghost process. A ghost process is one that exists in the agent's inventory but is not present in the OR state update.
This is happening due to out of order messages. The OR generates its state updates at regular intervals (every 20 secs) and sends them to a jms topic. In this case, the PM fetches the update, trims it and sends it to agents. Not 100% sure, but it looks like there is some kind of a delay in a broker which causes messages to be held and than subsequently delivered in rapid succession.
The agent should be able to cope with this. The OR should add a sequence number to each state it publishes. The PM should forward this sequence when it sends a trimmed state to the agents. The agent should compare received sequence against what it received on the previous update.If the new sequence is larger than the previous one, processing should be allowed. If the new sequence is smaller then previous, the message should be discarded.