Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Won't Do
-
1.6
-
None
-
None
-
Ubuntu 14.04
Description
Steps to reproduce:
1. In Terminal A, start ApolloMQ via apollo-cli.
2. In Terminal B, do the same thing with the same options from the same working directory. Notice the log message: `WARN | DB operation failed. (entering recovery mode): java.io.IOException: File 'message-broker/data/lock' could not be locked.`
3. In Terminal C, kill (SIGTERM) the ApolloMQ you created in Terminal B.
Expected results: the process from Terminal B dies.
Actual results: the process from Terminal B stays open, and only a SIGKILL can kill it. That's because the shutdown hook waits for the broker to stop, but the broker won't stop because it never started.
Apollo's shutdown hook is a huge sysadmin headache for us. It hurts us in two places:
A) On our cloud server, Upstart sends a SIGTERM and waits before sending a final SIGKILL. That means it takes an excessive amount of time for us to restart Apollo: we need to "stop; sleep 6; start" or we can't know the restart will succeed. Sure, we can solve this with extra sysadminning that sends SIGKILL judiciously; but if we're supposed to SIGKILL Apollo, then why is there a shutdown hook in the first place?
B) We distribute a download-and-run parent program that invokes and monitors Apollo. This is a cross-platform Java program, so it can only send a SIGTERM. If Apollo doesn't shut down from a SIGTERM, it stays running forever, and it blocks new Apollo processes from starting.