ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-1264

FollowerResyncConcurrencyTest failing intermittently

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.3.3, 3.4.0, 3.5.0
    • Fix Version/s: 3.3.4, 3.4.0, 3.5.0
    • Component/s: tests
    • Labels:
      None
    • Release Note:
      Revision 1198053

      Description

      The FollowerResyncConcurrencyTest test is failing intermittently.

      saw the following on 3.4:

      junit.framework.AssertionFailedError: Should have same number of
      ephemerals in both followers expected:<11741> but was:<14001>
             at org.apache.zookeeper.test.FollowerResyncConcurrencyTest.verifyState(FollowerResyncConcurrencyTest.java:400)
             at org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testResyncBySnapThenDiffAfterFollowerCrashes(FollowerResyncConcurrencyTest.java:196)
             at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
      
      1. ZOOKEEPER-1264_branch33.patch
        13 kB
        Patrick Hunt
      2. ZOOKEEPER-1264_branch34.patch
        13 kB
        Patrick Hunt
      3. ZOOKEEPER-1264.patch
        13 kB
        Patrick Hunt
      4. followerresyncfailure_log.txt.gz
        36 kB
        Patrick Hunt
      5. logs.zip
        35 kB
        Patrick Hunt
      6. tmp.zip
        1.11 MB
        Patrick Hunt
      7. ZOOKEEPER-1264.patch
        4 kB
        Camille Fournier
      8. ZOOKEEPER-1264unittest.patch
        13 kB
        Benjamin Reed
      9. ZOOKEEPER-1264unittest.patch
        14 kB
        Benjamin Reed
      10. ZOOKEEPER-1264.patch
        16 kB
        Benjamin Reed
      11. ZOOKEEPER-1264-merge.patch
        19 kB
        Camille Fournier
      12. ZOOKEEPER-1264.patch
        22 kB
        Benjamin Reed
      13. ZOOKEEPER-1264-branch34.patch
        26 kB
        Camille Fournier
      14. ZOOKEEPER-1264-34-bad.patch
        23 kB
        Camille Fournier
      15. ZOOKEEPER-1264-latest.patch
        23 kB
        Camille Fournier
      16. ZOOKEEPER-ACCEPTEDEPOCH.patch
        1 kB
        Alexander Shraer
      17. ZOOKEEPER-ACCEPTEDEPOCH-trunk.patch
        1 kB
        Alexander Shraer
      18. ZOOKEEPER-1264-br34-final.patch
        25 kB
        Camille Fournier
      19. ZOOKEEPER-1264-final.patch
        24 kB
        Camille Fournier

        Issue Links

          Activity

          Hide
          Patrick Hunt added a comment -

          test is using 1 second session timeout. I'll cleanup the test a bit and verify.

          Show
          Patrick Hunt added a comment - test is using 1 second session timeout. I'll cleanup the test a bit and verify.
          Hide
          Patrick Hunt added a comment -

          these patches fix the problem for me (verified each patch, but only trunk run against CI host having issues originally - three runs, each time passed)

          Main changes are cleaned up generally, fixed session timeout, cleaned up test clients/quorum, etc...

          Show
          Patrick Hunt added a comment - these patches fix the problem for me (verified each patch, but only trunk run against CI host having issues originally - three runs, each time passed) Main changes are cleaned up generally, fixed session timeout, cleaned up test clients/quorum, etc...
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12501240/ZOOKEEPER-1264.patch
          against trunk revision 1190073.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/709//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/709//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/709//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501240/ZOOKEEPER-1264.patch against trunk revision 1190073. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/709//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/709//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/709//console This message is automatically generated.
          Hide
          Mahadev konar added a comment -

          +1 looks good to me. Might want to check on the the hudson tests. Looks like https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/709//testReport/ has observer test failing? Doesnt seem related but no harm in running the trunk patch through hudson again.

          Show
          Mahadev konar added a comment - +1 looks good to me. Might want to check on the the hudson tests. Looks like https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/709//testReport/ has observer test failing? Doesnt seem related but no harm in running the trunk patch through hudson again.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12501240/ZOOKEEPER-1264.patch
          against trunk revision 1190073.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/710//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/710//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/710//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501240/ZOOKEEPER-1264.patch against trunk revision 1190073. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/710//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/710//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/710//console This message is automatically generated.
          Hide
          Camille Fournier added a comment -

          This looks like a good cleanup, thanks Patrick.

          Show
          Camille Fournier added a comment - This looks like a good cleanup, thanks Patrick.
          Hide
          Camille Fournier added a comment -

          Committed to trunk, 3.3.4 and 3.4 branches.

          Show
          Camille Fournier added a comment - Committed to trunk, 3.3.4 and 3.4 branches.
          Hide
          Patrick Hunt added a comment -

          I'm afraid this patch has not entirely fixed the problem. After 6 runs on my CI host it failed again. Perhaps Camille can take a look - I'll attach a log file momentarily.

          Show
          Patrick Hunt added a comment - I'm afraid this patch has not entirely fixed the problem. After 6 runs on my CI host it failed again. Perhaps Camille can take a look - I'll attach a log file momentarily.
          Hide
          Patrick Hunt added a comment -

          most recent failure with original patch applied.

          Show
          Patrick Hunt added a comment - most recent failure with original patch applied.
          Hide
          Camille Fournier added a comment -

          Looking.

          Show
          Camille Fournier added a comment - Looking.
          Hide
          Camille Fournier added a comment -

          It might also be somewhat helpful if you could send me the txn logs from the test servers but I realize that might be too much to ask.

          Show
          Camille Fournier added a comment - It might also be somewhat helpful if you could send me the txn logs from the test servers but I realize that might be too much to ask.
          Hide
          Patrick Hunt added a comment -

          great idea, those are lost to the ages but I'll try to reproduce.

          Show
          Patrick Hunt added a comment - great idea, those are lost to the ages but I'll try to reproduce.
          Hide
          Camille Fournier added a comment -

          Yeah, I spent a bit of time looking at this. I have a few ideas but it would probably go a lot faster if I had logs to examine since I can't seem to repro it myself. If you can get me some I will look more this weekend.

          Show
          Camille Fournier added a comment - Yeah, I spent a bit of time looking at this. I have a few ideas but it would probably go a lot faster if I had logs to examine since I can't seem to repro it myself. If you can get me some I will look more this weekend.
          Hide
          Patrick Hunt added a comment -

          Your wish == my command.

          See tmp and log attachments for a sample run that fails.

          I can easily reproduce this, so ff to ping me if you want to try a fix (attach to the jira).

          Show
          Patrick Hunt added a comment - Your wish == my command. See tmp and log attachments for a sample run that fails. I can easily reproduce this, so ff to ping me if you want to try a fix (attach to the jira).
          Hide
          Camille Fournier added a comment -

          Thanks Patrick. My suspicions were true, the failing zk has a chunk missing out of its logs that corresponds to the missing ephemeral nodes (snapshot snapshot.100002322, log log.100002c3e, but the earlier log file doesn't have txns between 2322 and 2c3e, they seem to just be missing). Now to figure out why it doesn't have those log files...

          Show
          Camille Fournier added a comment - Thanks Patrick. My suspicions were true, the failing zk has a chunk missing out of its logs that corresponds to the missing ephemeral nodes (snapshot snapshot.100002322, log log.100002c3e, but the earlier log file doesn't have txns between 2322 and 2c3e, they seem to just be missing). Now to figure out why it doesn't have those log files...
          Hide
          Hudson added a comment -

          Integrated in ZooKeeper-trunk #1348 (See https://builds.apache.org/job/ZooKeeper-trunk/1348/)
          ZOOKEEPER-1264: FollowerResyncConcurrencyTest failing intermittently (phunt via camille)

          camille : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190358
          Files :

          • /zookeeper/trunk/CHANGES.txt
          • /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/FollowerResyncConcurrencyTest.java
          • /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/QuorumTest.java
          Show
          Hudson added a comment - Integrated in ZooKeeper-trunk #1348 (See https://builds.apache.org/job/ZooKeeper-trunk/1348/ ) ZOOKEEPER-1264 : FollowerResyncConcurrencyTest failing intermittently (phunt via camille) camille : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190358 Files : /zookeeper/trunk/CHANGES.txt /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/FollowerResyncConcurrencyTest.java /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/QuorumTest.java
          Hide
          Camille Fournier added a comment -

          Got this reproduced on my local box with yet more hacks to the test and a few sleeps in the source code. Should be close to figuring out the problem, probably tomorrow sometime. Stay tuned.

          Show
          Camille Fournier added a comment - Got this reproduced on my local box with yet more hacks to the test and a few sleeps in the source code. Should be close to figuring out the problem, probably tomorrow sometime. Stay tuned.
          Hide
          Camille Fournier added a comment -

          OK, I found the bug. Ben, we could use your attention here.

          The problem is that we queue NEWLEADER before we queue UPTODATE, but inbetween these messages we send more sync packets to move us from SNAP to, well, UPTODATE. These get written directly to the data tree, bypassing the log. But if you immediately shut down the ZK before snapshotting again, you will lose any record of these transactions on the ZK in question. It seems to me that we should either snapshot again on UPTODATE or else wait to snapshot in the first place until that packet is sent. I don't understand why we moved to snapshot on NEWLEADER in the first place. If one of the ZAB 1.0 authors could comment, that would be useful.

          Show
          Camille Fournier added a comment - OK, I found the bug. Ben, we could use your attention here. The problem is that we queue NEWLEADER before we queue UPTODATE, but inbetween these messages we send more sync packets to move us from SNAP to, well, UPTODATE. These get written directly to the data tree, bypassing the log. But if you immediately shut down the ZK before snapshotting again, you will lose any record of these transactions on the ZK in question. It seems to me that we should either snapshot again on UPTODATE or else wait to snapshot in the first place until that packet is sent. I don't understand why we moved to snapshot on NEWLEADER in the first place. If one of the ZAB 1.0 authors could comment, that would be useful.
          Hide
          Mahadev konar added a comment -

          Ben/Flavio,
          Any comments?

          Show
          Mahadev konar added a comment - Ben/Flavio, Any comments?
          Hide
          Camille Fournier added a comment -

          From a comment I added to the tracker that this change was attached to:
          ZOOKEEPER-1136 causes a concurrency bug. Specifically:
          1. Follower rejoins, gets snap from leader
          2. Follower gets NEWLEADER message and takes a snapshot
          3. Follower gets some additional tranactions forwarded from leader, applies these directly to data tree
          4. Follower gets an UPTODATE message, does not take a snapshot
          5. Follower starts following, writes some new transactions to its log, and is killed before it takes another snapshot
          6. Follower restarts and gets a DIFF from the leader

          The transactions that came in between NEWLEADER and UPTODATE are lost because they never go anywhere but the internal data tree, and if that tree isn't snapshotted and the follower restarts with only a DIFF, the follower will lose these transactions.

          I think the proper thing to do is snapshot after UPTODATE, but I'm not sure why we changed this to snapshot after NEWLEADER instead. The wiki doesn't seem to explain that clearly.

          Show
          Camille Fournier added a comment - From a comment I added to the tracker that this change was attached to: ZOOKEEPER-1136 causes a concurrency bug. Specifically: 1. Follower rejoins, gets snap from leader 2. Follower gets NEWLEADER message and takes a snapshot 3. Follower gets some additional tranactions forwarded from leader, applies these directly to data tree 4. Follower gets an UPTODATE message, does not take a snapshot 5. Follower starts following, writes some new transactions to its log, and is killed before it takes another snapshot 6. Follower restarts and gets a DIFF from the leader The transactions that came in between NEWLEADER and UPTODATE are lost because they never go anywhere but the internal data tree, and if that tree isn't snapshotted and the follower restarts with only a DIFF, the follower will lose these transactions. I think the proper thing to do is snapshot after UPTODATE, but I'm not sure why we changed this to snapshot after NEWLEADER instead. The wiki doesn't seem to explain that clearly.
          Hide
          Benjamin Reed added a comment -

          the reason we take a snapshot after NEWLEADER is because the follower is promising to reproduce the state that it received from the leader at that point. phase 2, step 2 in the zab 1.0 on the wiki. the transactions that follow NEWLEADER should be logged to disk using normal transaction logging. i'll take a look at the code.

          Show
          Benjamin Reed added a comment - the reason we take a snapshot after NEWLEADER is because the follower is promising to reproduce the state that it received from the leader at that point. phase 2, step 2 in the zab 1.0 on the wiki. the transactions that follow NEWLEADER should be logged to disk using normal transaction logging. i'll take a look at the code.
          Hide
          Camille Fournier added a comment -

          Thanks Ben. The patch I attached changes both Learner and FollowerResyncConcurrencyTest. You should be able to repro the failure with testResyncBySnapThenDiffAfterFollowerCrashes pretty reliably. You can ignore the changes in Learner (just move the snap to after UPTODATE instead of NEWLEADER0.

          Show
          Camille Fournier added a comment - Thanks Ben. The patch I attached changes both Learner and FollowerResyncConcurrencyTest. You should be able to repro the failure with testResyncBySnapThenDiffAfterFollowerCrashes pretty reliably. You can ignore the changes in Learner (just move the snap to after UPTODATE instead of NEWLEADER0.
          Hide
          Patrick Hunt added a comment -

          Awesome job Camille, thanks!

          Show
          Patrick Hunt added a comment - Awesome job Camille, thanks!
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12501785/ZOOKEEPER-1264.patch
          against trunk revision 1196025.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/754//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/754//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/754//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501785/ZOOKEEPER-1264.patch against trunk revision 1196025. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/754//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/754//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/754//console This message is automatically generated.
          Hide
          Benjamin Reed added a comment -

          ah cool. ok i'll check it out now.

          Show
          Benjamin Reed added a comment - ah cool. ok i'll check it out now.
          Hide
          Benjamin Reed added a comment -

          i applied your patch to trunk and i removed the change to Learner and the test passes is it an intermittent failure?

          Show
          Benjamin Reed added a comment - i applied your patch to trunk and i removed the change to Learner and the test passes is it an intermittent failure?
          Hide
          Camille Fournier added a comment -

          Yeah, sorry, these concurrency tests are pretty much impossible to write deterministically without some additional scaffolding. If you look at lines 152-158 of the test, you want the thread that I started to have transactions passing through the leader when the qu.restart at 153 loads the follower. The follower should get a snapshot from the leader, a few more pending transactions, and then additional transactions that cause a log file to be written that will have a zxid that is not the zxid of the snapshot it created + 1. For example from Pat's log:
          2011-10-28 17:09:56,691 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11221:FileTxnSnapLog@255] - Snapshotting: 100002322

          (indicating the NEWLEADER)
          then

          2011-10-28 17:09:59,316 [myid:] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11221:Follower@118] - Got zxid 0x100002c3e expected 0x1
          2011-10-28 17:09:59,330 [myid:] - INFO [SyncThread:1:FileTxnLog@195] - Creating new log file: log.100002c3e

          Show
          Camille Fournier added a comment - Yeah, sorry, these concurrency tests are pretty much impossible to write deterministically without some additional scaffolding. If you look at lines 152-158 of the test, you want the thread that I started to have transactions passing through the leader when the qu.restart at 153 loads the follower. The follower should get a snapshot from the leader, a few more pending transactions, and then additional transactions that cause a log file to be written that will have a zxid that is not the zxid of the snapshot it created + 1. For example from Pat's log: 2011-10-28 17:09:56,691 [myid:] - INFO [QuorumPeer [myid=1] /0:0:0:0:0:0:0:0:11221:FileTxnSnapLog@255] - Snapshotting: 100002322 (indicating the NEWLEADER) then 2011-10-28 17:09:59,316 [myid:] - WARN [QuorumPeer [myid=1] /0:0:0:0:0:0:0:0:11221:Follower@118] - Got zxid 0x100002c3e expected 0x1 2011-10-28 17:09:59,330 [myid:] - INFO [SyncThread:1:FileTxnLog@195] - Creating new log file: log.100002c3e
          Hide
          Benjamin Reed added a comment -

          got it. i think i see how to make this reproduce more reliable. i'll be back

          Show
          Benjamin Reed added a comment - got it. i think i see how to make this reproduce more reliable. i'll be back
          Hide
          Mahadev konar added a comment -

          @Ben,
          sorry to be pestering, I'd like to get 3.4 rc1 out today. Please be back today .

          Show
          Mahadev konar added a comment - @Ben, sorry to be pestering, I'd like to get 3.4 rc1 out today. Please be back today .
          Hide
          Mahadev konar added a comment -

          @Ben,
          Any update?

          Show
          Mahadev konar added a comment - @Ben, Any update?
          Hide
          Benjamin Reed added a comment -

          i've created a unit test to reproduce the problem since we can test it more directly and deterministicly, but i can't seem to make it happen. i'm attaching my unit test patch just in case you are camille can see what i'm missing.

          if i understand it, the problem is that we are losing proposals that are received between the NEWLEADER and the UPDATE, but a follower doesn't send out any acks during that time, so it's okay to lose them. am i misunderstanding the problem?

          Show
          Benjamin Reed added a comment - i've created a unit test to reproduce the problem since we can test it more directly and deterministicly, but i can't seem to make it happen. i'm attaching my unit test patch just in case you are camille can see what i'm missing. if i understand it, the problem is that we are losing proposals that are received between the NEWLEADER and the UPDATE, but a follower doesn't send out any acks during that time, so it's okay to lose them. am i misunderstanding the problem?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12501918/ZOOKEEPER-1264unittest.patch
          against trunk revision 1196025.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/760//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/760//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/760//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501918/ZOOKEEPER-1264unittest.patch against trunk revision 1196025. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/760//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/760//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/760//console This message is automatically generated.
          Hide
          Benjamin Reed added a comment -

          just a quick update. i'm able to reproduce the problem quickly and deterministicly with a unit test. it has to do with the snapshot transfer. it is not getting recorded with the correct zxid. i'll have the fix shortly.

          Show
          Benjamin Reed added a comment - just a quick update. i'm able to reproduce the problem quickly and deterministicly with a unit test. it has to do with the snapshot transfer. it is not getting recorded with the correct zxid. i'll have the fix shortly.
          Hide
          Benjamin Reed added a comment -

          here is the unit test. doing a snapshot at UPDATE will make this test pass, but i'm afraid it is masking a deeper problem. the question is, why does it fix the problem?

          Show
          Benjamin Reed added a comment - here is the unit test. doing a snapshot at UPDATE will make this test pass, but i'm afraid it is masking a deeper problem. the question is, why does it fix the problem?
          Hide
          Camille Fournier added a comment -

          Because when the follower writes a new log file without writing a snapshot with the old transactions, on restart the ZK thinks it has the transactions up to the zxid in the log file. The fact that these transactions were never written to a log or snapshot by the follower is not captured. We got a NEWLEADER and took a snapshot, then got a bunch of txns that went directly to our data tree, then got UPTODATE, then some other new transactions that caused the creation of a brand new log file. The intermediate transactions between NEWLEADER and UPTODATE are never written to a persistent store on the follower unless it manages to stay alive long enough to do another snapshot.

          Show
          Camille Fournier added a comment - Because when the follower writes a new log file without writing a snapshot with the old transactions, on restart the ZK thinks it has the transactions up to the zxid in the log file. The fact that these transactions were never written to a log or snapshot by the follower is not captured. We got a NEWLEADER and took a snapshot, then got a bunch of txns that went directly to our data tree, then got UPTODATE, then some other new transactions that caused the creation of a brand new log file. The intermediate transactions between NEWLEADER and UPTODATE are never written to a persistent store on the follower unless it manages to stay alive long enough to do another snapshot.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12501973/ZOOKEEPER-1264unittest.patch
          against trunk revision 1196025.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/766//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/766//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/766//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501973/ZOOKEEPER-1264unittest.patch against trunk revision 1196025. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/766//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/766//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/766//console This message is automatically generated.
          Hide
          Benjamin Reed added a comment -

          i think i got it. camille can you try it with your test to see if it's fixed there as well? (the tests always passed on my machine.)

          Show
          Benjamin Reed added a comment - i think i got it. camille can you try it with your test to see if it's fixed there as well? (the tests always passed on my machine.)
          Hide
          Camille Fournier added a comment -

          Seems to work. I want to go ahead and put in the additional changes to FollowerResyncConcurrencyTest along with your patch after I finish reviewing it. Theoretically they aren't needed but given how many times this test has caught issues I think it's worth it to double-test this stuff. Let me know if you disagree.

          Show
          Camille Fournier added a comment - Seems to work. I want to go ahead and put in the additional changes to FollowerResyncConcurrencyTest along with your patch after I finish reviewing it. Theoretically they aren't needed but given how many times this test has caught issues I think it's worth it to double-test this stuff. Let me know if you disagree.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12501981/ZOOKEEPER-1264.patch
          against trunk revision 1196025.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/767//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/767//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/767//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501981/ZOOKEEPER-1264.patch against trunk revision 1196025. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/767//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/767//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/767//console This message is automatically generated.
          Hide
          Benjamin Reed added a comment -

          i agree, we should have the extra test. it is nice to have the extra coverage. do you want to generate a merged patch?

          Show
          Benjamin Reed added a comment - i agree, we should have the extra test. it is nice to have the extra coverage. do you want to generate a merged patch?
          Hide
          Camille Fournier added a comment -

          Yup will do asap (which might be early this evening but I'll try to get it in a few mins).

          Show
          Camille Fournier added a comment - Yup will do asap (which might be early this evening but I'll try to get it in a few mins).
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12502007/ZOOKEEPER-1264-merge.patch
          against trunk revision 1196025.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/768//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/768//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/768//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502007/ZOOKEEPER-1264-merge.patch against trunk revision 1196025. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/768//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/768//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/768//console This message is automatically generated.
          Hide
          Benjamin Reed added a comment -

          this patch merges camille's test in as well. it also adds a couple of extra asserts to cover ZOOKEEPER-1282. finally, it also moves around a couple of lines to fix ZOOKEEPER-1282. (i merged in 1282 because the fix and tests were simple modifications of this patch and we need to get this out asap.)

          Show
          Benjamin Reed added a comment - this patch merges camille's test in as well. it also adds a couple of extra asserts to cover ZOOKEEPER-1282 . finally, it also moves around a couple of lines to fix ZOOKEEPER-1282 . (i merged in 1282 because the fix and tests were simple modifications of this patch and we need to get this out asap.)
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12502100/ZOOKEEPER-1264.patch
          against trunk revision 1196819.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/770//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/770//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/770//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502100/ZOOKEEPER-1264.patch against trunk revision 1196819. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/770//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/770//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/770//console This message is automatically generated.
          Hide
          Camille Fournier added a comment -

          Ben, just two questions:
          Does this logic really only apply to FollowerZookeeperServers or should observers also do this?

          Why does the playing of these txns to the log come after we start the zk server instead of before?

          Show
          Camille Fournier added a comment - Ben, just two questions: Does this logic really only apply to FollowerZookeeperServers or should observers also do this? Why does the playing of these txns to the log come after we start the zk server instead of before?
          Hide
          Benjamin Reed added a comment -

          yeah, the observers (as far as my understanding from the code) don't really log transactions and they only get INFORMs no PROPOSALs.

          we can start writing the transactions after the newleader message and i looked at doing that. the code just got a little to complicated for very little benefit. all the proposals up to the uptodate are already committed, so we aren't waiting on an ack. and we would have to move to writing out the pending proposals until after the newleader or uptodate (for backwards compatibility) and then check if have already started writing in the PROPOSAL handling. it just seemed simpler to wait until the end and do it all at once.

          Show
          Benjamin Reed added a comment - yeah, the observers (as far as my understanding from the code) don't really log transactions and they only get INFORMs no PROPOSALs. we can start writing the transactions after the newleader message and i looked at doing that. the code just got a little to complicated for very little benefit. all the proposals up to the uptodate are already committed, so we aren't waiting on an ack. and we would have to move to writing out the pending proposals until after the newleader or uptodate (for backwards compatibility) and then check if have already started writing in the PROPOSAL handling. it just seemed simpler to wait until the end and do it all at once.
          Hide
          Benjamin Reed added a comment -

          henry could you check the observer handling?

          Show
          Benjamin Reed added a comment - henry could you check the observer handling?
          Hide
          Camille Fournier added a comment -

          Ok, I think this is all fine. I will check this in.

          Show
          Camille Fournier added a comment - Ok, I think this is all fine. I will check this in.
          Hide
          Laxman added a comment -

          Similar issue occurred in our case as well. The ephemeral exists only in one of the follower even after it's owner session is expired. I wanted to know the exact scenario when this may happen, so that I can add a testcase to verify.

          Show
          Laxman added a comment - Similar issue occurred in our case as well. The ephemeral exists only in one of the follower even after it's owner session is expired. I wanted to know the exact scenario when this may happen, so that I can add a testcase to verify.
          Hide
          Camille Fournier added a comment -

          Can someone take a look at this patch? I made it last night to check in to the 3.4 branch but the Zab1_0Test fails when I run it.

          Show
          Camille Fournier added a comment - Can someone take a look at this patch? I made it last night to check in to the 3.4 branch but the Zab1_0Test fails when I run it.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12502472/ZOOKEEPER-1264-branch34.patch
          against trunk revision 1196819.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/775//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502472/ZOOKEEPER-1264-branch34.patch against trunk revision 1196819. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/775//console This message is automatically generated.
          Hide
          Mahadev konar added a comment -

          Camille,
          Are you debugging the test failure in 3.4 or waiting for others to take a look?

          Show
          Mahadev konar added a comment - Camille, Are you debugging the test failure in 3.4 or waiting for others to take a look?
          Hide
          Camille Fournier added a comment -

          I'm looking at it. The test that fails is not a new test, it's testAbandonBeforeACKEpoch. I have no idea why it is failing.

          Show
          Camille Fournier added a comment - I'm looking at it. The test that fails is not a new test, it's testAbandonBeforeACKEpoch. I have no idea why it is failing.
          Hide
          Camille Fournier added a comment -

          Oh now I see. Because 1192 introduced fixes into leader election that added stuff to the Zab1_0Test that I missed. Why in the world do we have leader election bugs going only into trunk instead of into 3.4 as well??? Not good.

          Show
          Camille Fournier added a comment - Oh now I see. Because 1192 introduced fixes into leader election that added stuff to the Zab1_0Test that I missed. Why in the world do we have leader election bugs going only into trunk instead of into 3.4 as well??? Not good.
          Hide
          Camille Fournier added a comment -

          I still cannot manage a clean patch of this issue and tests to the 3.4 branch. This is my latest attempt, which still fails the Zab1_0Test. Can someone else please take a look here?

          Show
          Camille Fournier added a comment - I still cannot manage a clean patch of this issue and tests to the 3.4 branch. This is my latest attempt, which still fails the Zab1_0Test. Can someone else please take a look here?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12502600/ZOOKEEPER-1264-34-bad.patch
          against trunk revision 1197891.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/776//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502600/ZOOKEEPER-1264-34-bad.patch against trunk revision 1197891. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/776//console This message is automatically generated.
          Hide
          Flavio Junqueira added a comment -

          I'll have a look.

          Show
          Flavio Junqueira added a comment - I'll have a look.
          Hide
          Camille Fournier added a comment -

          This is the patch to trunk, verifying via hudson that it passes everything.

          Show
          Camille Fournier added a comment - This is the patch to trunk, verifying via hudson that it passes everything.
          Hide
          Camille Fournier added a comment -

          Ben added a check to testNormalRun at line 502:

          Assert.assertEquals(1, l.self.getAcceptedEpoch());

          This line consistently fails for me. I'm not sure why this would be failing in the 3.4 branch but not trunk.

          Show
          Camille Fournier added a comment - Ben added a check to testNormalRun at line 502: Assert.assertEquals(1, l.self.getAcceptedEpoch()); This line consistently fails for me. I'm not sure why this would be failing in the 3.4 branch but not trunk.
          Hide
          Alexander Shraer added a comment -

          I know where the problem is. Ben assumes that acceptedEpoch is set atomically before the leader replies to the client with LEADERINFO, but it doesn't. The leader sets it after returning from getEpochToPropose. This is a bug. I'll upload a small patch in a sec

          Show
          Alexander Shraer added a comment - I know where the problem is. Ben assumes that acceptedEpoch is set atomically before the leader replies to the client with LEADERINFO, but it doesn't. The leader sets it after returning from getEpochToPropose. This is a bug. I'll upload a small patch in a sec
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12502605/ZOOKEEPER-1264-latest.patch
          against trunk revision 1197891.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/777//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/777//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/777//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502605/ZOOKEEPER-1264-latest.patch against trunk revision 1197891. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/777//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/777//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/777//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12502606/ZOOKEEPER-ACCEPTEDEPOCH.patch
          against trunk revision 1197891.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/778//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502606/ZOOKEEPER-ACCEPTEDEPOCH.patch against trunk revision 1197891. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/778//console This message is automatically generated.
          Hide
          Flavio Junqueira added a comment -

          +1, looks right to me.

          Show
          Flavio Junqueira added a comment - +1, looks right to me.
          Hide
          Camille Fournier added a comment -

          Final version of fix for 3.4 branch

          Show
          Camille Fournier added a comment - Final version of fix for 3.4 branch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12502608/ZOOKEEPER-1264-br34-final.patch
          against trunk revision 1197891.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/779//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502608/ZOOKEEPER-1264-br34-final.patch against trunk revision 1197891. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/779//console This message is automatically generated.
          Hide
          Camille Fournier added a comment -

          final trunk patch

          Show
          Camille Fournier added a comment - final trunk patch
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12502617/ZOOKEEPER-1264-final.patch
          against trunk revision 1197891.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/781//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/781//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/781//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502617/ZOOKEEPER-1264-final.patch against trunk revision 1197891. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/781//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/781//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/781//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12502617/ZOOKEEPER-1264-final.patch
          against trunk revision 1197891.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/782//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/782//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/782//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502617/ZOOKEEPER-1264-final.patch against trunk revision 1197891. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/782//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/782//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/782//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in ZooKeeper-trunk #1357 (See https://builds.apache.org/job/ZooKeeper-trunk/1357/)
          ZOOKEEPER-1264. FollowerResyncConcurrencyTest failing intermittently.
          ZOOKEEPER-1282. Learner.java not following Zab 1.0 protocol - setCurrentEpoch should be done upon receipt of NEWLEADER (before acking it) and not upon receipt of UPTODATE.
          ZOOKEEPER-1291. AcceptedEpoch not updated at leader before it proposes the epoch to followers.

          camille : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1198053
          Files :

          • /zookeeper/trunk/CHANGES.txt
          • /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/Leader.java
          • /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/Learner.java
          • /zookeeper/trunk/src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java
          • /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/FollowerResyncConcurrencyTest.java
          Show
          Hudson added a comment - Integrated in ZooKeeper-trunk #1357 (See https://builds.apache.org/job/ZooKeeper-trunk/1357/ ) ZOOKEEPER-1264 . FollowerResyncConcurrencyTest failing intermittently. ZOOKEEPER-1282 . Learner.java not following Zab 1.0 protocol - setCurrentEpoch should be done upon receipt of NEWLEADER (before acking it) and not upon receipt of UPTODATE. ZOOKEEPER-1291 . AcceptedEpoch not updated at leader before it proposes the epoch to followers. camille : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1198053 Files : /zookeeper/trunk/CHANGES.txt /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/Leader.java /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/Learner.java /zookeeper/trunk/src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/FollowerResyncConcurrencyTest.java

            People

            • Assignee:
              Camille Fournier
              Reporter:
              Patrick Hunt
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development