this is a patch that addresses the newleader logging issue, and brings the implementation in line with the theoretically proven protocol.

there is a problem with hierarchal quorums that i haven't looked at. the backwards compatibility code is in, but i haven't tested it. i'm still trying to make leader election mod in a backward compatible way.

the code LearnerHandler code is pretty messy now and i'm torn between cleaning it up or leaving until later. my concerns about cleaning is that 1) it may be hard to compare past logic with current if we need to in the near future and 2) it may introduce new bugs. on the pro side, it is crying out for the code cleanup, and this is usually when we do it.

oh btw, the test case was super hard to write. the scenario is easy to describe, but i found it quite hard to implement. if someone (vishal? sees a better way to do the test, i would be really happy to try it.

Hi Ben,

I have attached the diff that I used to repro the bug.

I inserted a failure in ProposalRequestProcessor.java. Everytime I read file /etc/zookeeper/debug.property. I check if the debug property is set. If it is, then I fail (I did shutdown -h to avoid some timing issues specific to our application) the leader after call to syncProcessor.processRequest(request). This ensures that the transaction is logged but not sent to followers. Then, when we restart the old leader we can check if it still has the old data.

Do you think this helps? I think we need to start injecting faults in the code itself. So having a framwork that would do something similar to the attached diff would be useful.

yeah, we really should be injecting faults. i think there are nice fault injectors that you can use without modifying your code right? have you (or anyone else listening) ever tried them?

Would mockito be helpful?

I dunno that mockito would do enough, but some of my colleagues had good luck with MultiThreadedTestCase:

http://code.google.com/p/multithreadedtc/

I didn't use it myself but it looked pretty cool.

Integrated in ZooKeeper-trunk #1215 (See https://builds.apache.org/job/ZooKeeper-trunk/1215/)