Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-1264

FollowerResyncConcurrencyTest failing intermittently

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 3.3.3, 3.4.0, 3.5.0
    • 3.3.4, 3.4.0, 3.5.0
    • tests
    • None
    • Revision 1198053

    Description

      The FollowerResyncConcurrencyTest test is failing intermittently.

      saw the following on 3.4:

      junit.framework.AssertionFailedError: Should have same number of
      ephemerals in both followers expected:<11741> but was:<14001>
             at org.apache.zookeeper.test.FollowerResyncConcurrencyTest.verifyState(FollowerResyncConcurrencyTest.java:400)
             at org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testResyncBySnapThenDiffAfterFollowerCrashes(FollowerResyncConcurrencyTest.java:196)
             at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:52)
      

      Attachments

        1. followerresyncfailure_log.txt.gz
          36 kB
          Patrick D. Hunt
        2. logs.zip
          35 kB
          Patrick D. Hunt
        3. tmp.zip
          1.11 MB
          Patrick D. Hunt
        4. ZOOKEEPER-1264_branch33.patch
          13 kB
          Patrick D. Hunt
        5. ZOOKEEPER-1264_branch34.patch
          13 kB
          Patrick D. Hunt
        6. ZOOKEEPER-1264.patch
          22 kB
          Benjamin Reed
        7. ZOOKEEPER-1264.patch
          16 kB
          Benjamin Reed
        8. ZOOKEEPER-1264.patch
          4 kB
          Camille Fournier
        9. ZOOKEEPER-1264.patch
          13 kB
          Patrick D. Hunt
        10. ZOOKEEPER-1264-34-bad.patch
          23 kB
          Camille Fournier
        11. ZOOKEEPER-1264-br34-final.patch
          25 kB
          Camille Fournier
        12. ZOOKEEPER-1264-branch34.patch
          26 kB
          Camille Fournier
        13. ZOOKEEPER-1264-final.patch
          24 kB
          Camille Fournier
        14. ZOOKEEPER-1264-latest.patch
          23 kB
          Camille Fournier
        15. ZOOKEEPER-1264-merge.patch
          19 kB
          Camille Fournier
        16. ZOOKEEPER-1264unittest.patch
          14 kB
          Benjamin Reed
        17. ZOOKEEPER-1264unittest.patch
          13 kB
          Benjamin Reed
        18. ZOOKEEPER-ACCEPTEDEPOCH.patch
          1 kB
          Alexander Shraer
        19. ZOOKEEPER-ACCEPTEDEPOCH-trunk.patch
          1 kB
          Alexander Shraer

        Issue Links

          Activity

            test is using 1 second session timeout. I'll cleanup the test a bit and verify.

            phunt Patrick D. Hunt added a comment - test is using 1 second session timeout. I'll cleanup the test a bit and verify.

            these patches fix the problem for me (verified each patch, but only trunk run against CI host having issues originally - three runs, each time passed)

            Main changes are cleaned up generally, fixed session timeout, cleaned up test clients/quorum, etc...

            phunt Patrick D. Hunt added a comment - these patches fix the problem for me (verified each patch, but only trunk run against CI host having issues originally - three runs, each time passed) Main changes are cleaned up generally, fixed session timeout, cleaned up test clients/quorum, etc...
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12501240/ZOOKEEPER-1264.patch
            against trunk revision 1190073.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 6 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            -1 core tests. The patch failed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/709//testReport/
            Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/709//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/709//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501240/ZOOKEEPER-1264.patch against trunk revision 1190073. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/709//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/709//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/709//console This message is automatically generated.
            mahadev Mahadev Konar added a comment -

            +1 looks good to me. Might want to check on the the hudson tests. Looks like https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/709//testReport/ has observer test failing? Doesnt seem related but no harm in running the trunk patch through hudson again.

            mahadev Mahadev Konar added a comment - +1 looks good to me. Might want to check on the the hudson tests. Looks like https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/709//testReport/ has observer test failing? Doesnt seem related but no harm in running the trunk patch through hudson again.
            hadoopqa Hadoop QA added a comment -

            +1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12501240/ZOOKEEPER-1264.patch
            against trunk revision 1190073.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 6 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/710//testReport/
            Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/710//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/710//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501240/ZOOKEEPER-1264.patch against trunk revision 1190073. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/710//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/710//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/710//console This message is automatically generated.

            This looks like a good cleanup, thanks Patrick.

            fournc Camille Fournier added a comment - This looks like a good cleanup, thanks Patrick.

            Committed to trunk, 3.3.4 and 3.4 branches.

            fournc Camille Fournier added a comment - Committed to trunk, 3.3.4 and 3.4 branches.

            I'm afraid this patch has not entirely fixed the problem. After 6 runs on my CI host it failed again. Perhaps Camille can take a look - I'll attach a log file momentarily.

            phunt Patrick D. Hunt added a comment - I'm afraid this patch has not entirely fixed the problem. After 6 runs on my CI host it failed again. Perhaps Camille can take a look - I'll attach a log file momentarily.

            most recent failure with original patch applied.

            phunt Patrick D. Hunt added a comment - most recent failure with original patch applied.

            Looking.

            fournc Camille Fournier added a comment - Looking.

            It might also be somewhat helpful if you could send me the txn logs from the test servers but I realize that might be too much to ask.

            fournc Camille Fournier added a comment - It might also be somewhat helpful if you could send me the txn logs from the test servers but I realize that might be too much to ask.

            great idea, those are lost to the ages but I'll try to reproduce.

            phunt Patrick D. Hunt added a comment - great idea, those are lost to the ages but I'll try to reproduce.

            Yeah, I spent a bit of time looking at this. I have a few ideas but it would probably go a lot faster if I had logs to examine since I can't seem to repro it myself. If you can get me some I will look more this weekend.

            fournc Camille Fournier added a comment - Yeah, I spent a bit of time looking at this. I have a few ideas but it would probably go a lot faster if I had logs to examine since I can't seem to repro it myself. If you can get me some I will look more this weekend.

            Your wish == my command.

            See tmp and log attachments for a sample run that fails.

            I can easily reproduce this, so ff to ping me if you want to try a fix (attach to the jira).

            phunt Patrick D. Hunt added a comment - Your wish == my command. See tmp and log attachments for a sample run that fails. I can easily reproduce this, so ff to ping me if you want to try a fix (attach to the jira).

            Thanks Patrick. My suspicions were true, the failing zk has a chunk missing out of its logs that corresponds to the missing ephemeral nodes (snapshot snapshot.100002322, log log.100002c3e, but the earlier log file doesn't have txns between 2322 and 2c3e, they seem to just be missing). Now to figure out why it doesn't have those log files...

            fournc Camille Fournier added a comment - Thanks Patrick. My suspicions were true, the failing zk has a chunk missing out of its logs that corresponds to the missing ephemeral nodes (snapshot snapshot.100002322, log log.100002c3e, but the earlier log file doesn't have txns between 2322 and 2c3e, they seem to just be missing). Now to figure out why it doesn't have those log files...
            hudson Hudson added a comment -

            Integrated in ZooKeeper-trunk #1348 (See https://builds.apache.org/job/ZooKeeper-trunk/1348/)
            ZOOKEEPER-1264: FollowerResyncConcurrencyTest failing intermittently (phunt via camille)

            camille : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190358
            Files :

            • /zookeeper/trunk/CHANGES.txt
            • /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/FollowerResyncConcurrencyTest.java
            • /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/QuorumTest.java
            hudson Hudson added a comment - Integrated in ZooKeeper-trunk #1348 (See https://builds.apache.org/job/ZooKeeper-trunk/1348/ ) ZOOKEEPER-1264 : FollowerResyncConcurrencyTest failing intermittently (phunt via camille) camille : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190358 Files : /zookeeper/trunk/CHANGES.txt /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/FollowerResyncConcurrencyTest.java /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/QuorumTest.java

            Got this reproduced on my local box with yet more hacks to the test and a few sleeps in the source code. Should be close to figuring out the problem, probably tomorrow sometime. Stay tuned.

            fournc Camille Fournier added a comment - Got this reproduced on my local box with yet more hacks to the test and a few sleeps in the source code. Should be close to figuring out the problem, probably tomorrow sometime. Stay tuned.

            OK, I found the bug. Ben, we could use your attention here.

            The problem is that we queue NEWLEADER before we queue UPTODATE, but inbetween these messages we send more sync packets to move us from SNAP to, well, UPTODATE. These get written directly to the data tree, bypassing the log. But if you immediately shut down the ZK before snapshotting again, you will lose any record of these transactions on the ZK in question. It seems to me that we should either snapshot again on UPTODATE or else wait to snapshot in the first place until that packet is sent. I don't understand why we moved to snapshot on NEWLEADER in the first place. If one of the ZAB 1.0 authors could comment, that would be useful.

            fournc Camille Fournier added a comment - OK, I found the bug. Ben, we could use your attention here. The problem is that we queue NEWLEADER before we queue UPTODATE, but inbetween these messages we send more sync packets to move us from SNAP to, well, UPTODATE. These get written directly to the data tree, bypassing the log. But if you immediately shut down the ZK before snapshotting again, you will lose any record of these transactions on the ZK in question. It seems to me that we should either snapshot again on UPTODATE or else wait to snapshot in the first place until that packet is sent. I don't understand why we moved to snapshot on NEWLEADER in the first place. If one of the ZAB 1.0 authors could comment, that would be useful.
            mahadev Mahadev Konar added a comment -

            Ben/Flavio,
            Any comments?

            mahadev Mahadev Konar added a comment - Ben/Flavio, Any comments?

            From a comment I added to the tracker that this change was attached to:
            ZOOKEEPER-1136 causes a concurrency bug. Specifically:
            1. Follower rejoins, gets snap from leader
            2. Follower gets NEWLEADER message and takes a snapshot
            3. Follower gets some additional tranactions forwarded from leader, applies these directly to data tree
            4. Follower gets an UPTODATE message, does not take a snapshot
            5. Follower starts following, writes some new transactions to its log, and is killed before it takes another snapshot
            6. Follower restarts and gets a DIFF from the leader

            The transactions that came in between NEWLEADER and UPTODATE are lost because they never go anywhere but the internal data tree, and if that tree isn't snapshotted and the follower restarts with only a DIFF, the follower will lose these transactions.

            I think the proper thing to do is snapshot after UPTODATE, but I'm not sure why we changed this to snapshot after NEWLEADER instead. The wiki doesn't seem to explain that clearly.

            fournc Camille Fournier added a comment - From a comment I added to the tracker that this change was attached to: ZOOKEEPER-1136 causes a concurrency bug. Specifically: 1. Follower rejoins, gets snap from leader 2. Follower gets NEWLEADER message and takes a snapshot 3. Follower gets some additional tranactions forwarded from leader, applies these directly to data tree 4. Follower gets an UPTODATE message, does not take a snapshot 5. Follower starts following, writes some new transactions to its log, and is killed before it takes another snapshot 6. Follower restarts and gets a DIFF from the leader The transactions that came in between NEWLEADER and UPTODATE are lost because they never go anywhere but the internal data tree, and if that tree isn't snapshotted and the follower restarts with only a DIFF, the follower will lose these transactions. I think the proper thing to do is snapshot after UPTODATE, but I'm not sure why we changed this to snapshot after NEWLEADER instead. The wiki doesn't seem to explain that clearly.
            breed Benjamin Reed added a comment -

            the reason we take a snapshot after NEWLEADER is because the follower is promising to reproduce the state that it received from the leader at that point. phase 2, step 2 in the zab 1.0 on the wiki. the transactions that follow NEWLEADER should be logged to disk using normal transaction logging. i'll take a look at the code.

            breed Benjamin Reed added a comment - the reason we take a snapshot after NEWLEADER is because the follower is promising to reproduce the state that it received from the leader at that point. phase 2, step 2 in the zab 1.0 on the wiki. the transactions that follow NEWLEADER should be logged to disk using normal transaction logging. i'll take a look at the code.

            Thanks Ben. The patch I attached changes both Learner and FollowerResyncConcurrencyTest. You should be able to repro the failure with testResyncBySnapThenDiffAfterFollowerCrashes pretty reliably. You can ignore the changes in Learner (just move the snap to after UPTODATE instead of NEWLEADER0.

            fournc Camille Fournier added a comment - Thanks Ben. The patch I attached changes both Learner and FollowerResyncConcurrencyTest. You should be able to repro the failure with testResyncBySnapThenDiffAfterFollowerCrashes pretty reliably. You can ignore the changes in Learner (just move the snap to after UPTODATE instead of NEWLEADER0.

            Awesome job Camille, thanks!

            phunt Patrick D. Hunt added a comment - Awesome job Camille, thanks!
            hadoopqa Hadoop QA added a comment -

            +1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12501785/ZOOKEEPER-1264.patch
            against trunk revision 1196025.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/754//testReport/
            Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/754//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/754//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501785/ZOOKEEPER-1264.patch against trunk revision 1196025. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/754//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/754//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/754//console This message is automatically generated.
            breed Benjamin Reed added a comment -

            ah cool. ok i'll check it out now.

            breed Benjamin Reed added a comment - ah cool. ok i'll check it out now.
            breed Benjamin Reed added a comment -

            i applied your patch to trunk and i removed the change to Learner and the test passes is it an intermittent failure?

            breed Benjamin Reed added a comment - i applied your patch to trunk and i removed the change to Learner and the test passes is it an intermittent failure?

            Yeah, sorry, these concurrency tests are pretty much impossible to write deterministically without some additional scaffolding. If you look at lines 152-158 of the test, you want the thread that I started to have transactions passing through the leader when the qu.restart at 153 loads the follower. The follower should get a snapshot from the leader, a few more pending transactions, and then additional transactions that cause a log file to be written that will have a zxid that is not the zxid of the snapshot it created + 1. For example from Pat's log:
            2011-10-28 17:09:56,691 [myid:] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11221:FileTxnSnapLog@255] - Snapshotting: 100002322

            (indicating the NEWLEADER)
            then

            2011-10-28 17:09:59,316 [myid:] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11221:Follower@118] - Got zxid 0x100002c3e expected 0x1
            2011-10-28 17:09:59,330 [myid:] - INFO [SyncThread:1:FileTxnLog@195] - Creating new log file: log.100002c3e

            fournc Camille Fournier added a comment - Yeah, sorry, these concurrency tests are pretty much impossible to write deterministically without some additional scaffolding. If you look at lines 152-158 of the test, you want the thread that I started to have transactions passing through the leader when the qu.restart at 153 loads the follower. The follower should get a snapshot from the leader, a few more pending transactions, and then additional transactions that cause a log file to be written that will have a zxid that is not the zxid of the snapshot it created + 1. For example from Pat's log: 2011-10-28 17:09:56,691 [myid:] - INFO [QuorumPeer [myid=1] /0:0:0:0:0:0:0:0:11221:FileTxnSnapLog@255] - Snapshotting: 100002322 (indicating the NEWLEADER) then 2011-10-28 17:09:59,316 [myid:] - WARN [QuorumPeer [myid=1] /0:0:0:0:0:0:0:0:11221:Follower@118] - Got zxid 0x100002c3e expected 0x1 2011-10-28 17:09:59,330 [myid:] - INFO [SyncThread:1:FileTxnLog@195] - Creating new log file: log.100002c3e
            breed Benjamin Reed added a comment -

            got it. i think i see how to make this reproduce more reliable. i'll be back

            breed Benjamin Reed added a comment - got it. i think i see how to make this reproduce more reliable. i'll be back
            mahadev Mahadev Konar added a comment -

            @Ben,
            sorry to be pestering, I'd like to get 3.4 rc1 out today. Please be back today .

            mahadev Mahadev Konar added a comment - @Ben, sorry to be pestering, I'd like to get 3.4 rc1 out today. Please be back today .
            mahadev Mahadev Konar added a comment -

            @Ben,
            Any update?

            mahadev Mahadev Konar added a comment - @Ben, Any update?
            breed Benjamin Reed added a comment -

            i've created a unit test to reproduce the problem since we can test it more directly and deterministicly, but i can't seem to make it happen. i'm attaching my unit test patch just in case you are camille can see what i'm missing.

            if i understand it, the problem is that we are losing proposals that are received between the NEWLEADER and the UPDATE, but a follower doesn't send out any acks during that time, so it's okay to lose them. am i misunderstanding the problem?

            breed Benjamin Reed added a comment - i've created a unit test to reproduce the problem since we can test it more directly and deterministicly, but i can't seem to make it happen. i'm attaching my unit test patch just in case you are camille can see what i'm missing. if i understand it, the problem is that we are losing proposals that are received between the NEWLEADER and the UPDATE, but a follower doesn't send out any acks during that time, so it's okay to lose them. am i misunderstanding the problem?
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12501918/ZOOKEEPER-1264unittest.patch
            against trunk revision 1196025.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            -1 core tests. The patch failed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/760//testReport/
            Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/760//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/760//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501918/ZOOKEEPER-1264unittest.patch against trunk revision 1196025. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/760//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/760//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/760//console This message is automatically generated.
            breed Benjamin Reed added a comment -

            just a quick update. i'm able to reproduce the problem quickly and deterministicly with a unit test. it has to do with the snapshot transfer. it is not getting recorded with the correct zxid. i'll have the fix shortly.

            breed Benjamin Reed added a comment - just a quick update. i'm able to reproduce the problem quickly and deterministicly with a unit test. it has to do with the snapshot transfer. it is not getting recorded with the correct zxid. i'll have the fix shortly.
            breed Benjamin Reed added a comment -

            here is the unit test. doing a snapshot at UPDATE will make this test pass, but i'm afraid it is masking a deeper problem. the question is, why does it fix the problem?

            breed Benjamin Reed added a comment - here is the unit test. doing a snapshot at UPDATE will make this test pass, but i'm afraid it is masking a deeper problem. the question is, why does it fix the problem?

            Because when the follower writes a new log file without writing a snapshot with the old transactions, on restart the ZK thinks it has the transactions up to the zxid in the log file. The fact that these transactions were never written to a log or snapshot by the follower is not captured. We got a NEWLEADER and took a snapshot, then got a bunch of txns that went directly to our data tree, then got UPTODATE, then some other new transactions that caused the creation of a brand new log file. The intermediate transactions between NEWLEADER and UPTODATE are never written to a persistent store on the follower unless it manages to stay alive long enough to do another snapshot.

            fournc Camille Fournier added a comment - Because when the follower writes a new log file without writing a snapshot with the old transactions, on restart the ZK thinks it has the transactions up to the zxid in the log file. The fact that these transactions were never written to a log or snapshot by the follower is not captured. We got a NEWLEADER and took a snapshot, then got a bunch of txns that went directly to our data tree, then got UPTODATE, then some other new transactions that caused the creation of a brand new log file. The intermediate transactions between NEWLEADER and UPTODATE are never written to a persistent store on the follower unless it manages to stay alive long enough to do another snapshot.
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12501973/ZOOKEEPER-1264unittest.patch
            against trunk revision 1196025.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            -1 core tests. The patch failed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/766//testReport/
            Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/766//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/766//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501973/ZOOKEEPER-1264unittest.patch against trunk revision 1196025. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/766//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/766//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/766//console This message is automatically generated.
            breed Benjamin Reed added a comment -

            i think i got it. camille can you try it with your test to see if it's fixed there as well? (the tests always passed on my machine.)

            breed Benjamin Reed added a comment - i think i got it. camille can you try it with your test to see if it's fixed there as well? (the tests always passed on my machine.)

            Seems to work. I want to go ahead and put in the additional changes to FollowerResyncConcurrencyTest along with your patch after I finish reviewing it. Theoretically they aren't needed but given how many times this test has caught issues I think it's worth it to double-test this stuff. Let me know if you disagree.

            fournc Camille Fournier added a comment - Seems to work. I want to go ahead and put in the additional changes to FollowerResyncConcurrencyTest along with your patch after I finish reviewing it. Theoretically they aren't needed but given how many times this test has caught issues I think it's worth it to double-test this stuff. Let me know if you disagree.
            hadoopqa Hadoop QA added a comment -

            +1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12501981/ZOOKEEPER-1264.patch
            against trunk revision 1196025.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/767//testReport/
            Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/767//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/767//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501981/ZOOKEEPER-1264.patch against trunk revision 1196025. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/767//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/767//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/767//console This message is automatically generated.
            breed Benjamin Reed added a comment -

            i agree, we should have the extra test. it is nice to have the extra coverage. do you want to generate a merged patch?

            breed Benjamin Reed added a comment - i agree, we should have the extra test. it is nice to have the extra coverage. do you want to generate a merged patch?

            Yup will do asap (which might be early this evening but I'll try to get it in a few mins).

            fournc Camille Fournier added a comment - Yup will do asap (which might be early this evening but I'll try to get it in a few mins).
            hadoopqa Hadoop QA added a comment -

            +1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12502007/ZOOKEEPER-1264-merge.patch
            against trunk revision 1196025.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 6 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/768//testReport/
            Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/768//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/768//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502007/ZOOKEEPER-1264-merge.patch against trunk revision 1196025. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/768//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/768//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/768//console This message is automatically generated.
            breed Benjamin Reed added a comment -

            this patch merges camille's test in as well. it also adds a couple of extra asserts to cover ZOOKEEPER-1282. finally, it also moves around a couple of lines to fix ZOOKEEPER-1282. (i merged in 1282 because the fix and tests were simple modifications of this patch and we need to get this out asap.)

            breed Benjamin Reed added a comment - this patch merges camille's test in as well. it also adds a couple of extra asserts to cover ZOOKEEPER-1282 . finally, it also moves around a couple of lines to fix ZOOKEEPER-1282 . (i merged in 1282 because the fix and tests were simple modifications of this patch and we need to get this out asap.)
            hadoopqa Hadoop QA added a comment -

            +1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12502100/ZOOKEEPER-1264.patch
            against trunk revision 1196819.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 6 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/770//testReport/
            Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/770//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/770//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502100/ZOOKEEPER-1264.patch against trunk revision 1196819. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/770//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/770//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/770//console This message is automatically generated.

            Ben, just two questions:
            Does this logic really only apply to FollowerZookeeperServers or should observers also do this?

            Why does the playing of these txns to the log come after we start the zk server instead of before?

            fournc Camille Fournier added a comment - Ben, just two questions: Does this logic really only apply to FollowerZookeeperServers or should observers also do this? Why does the playing of these txns to the log come after we start the zk server instead of before?
            breed Benjamin Reed added a comment -

            yeah, the observers (as far as my understanding from the code) don't really log transactions and they only get INFORMs no PROPOSALs.

            we can start writing the transactions after the newleader message and i looked at doing that. the code just got a little to complicated for very little benefit. all the proposals up to the uptodate are already committed, so we aren't waiting on an ack. and we would have to move to writing out the pending proposals until after the newleader or uptodate (for backwards compatibility) and then check if have already started writing in the PROPOSAL handling. it just seemed simpler to wait until the end and do it all at once.

            breed Benjamin Reed added a comment - yeah, the observers (as far as my understanding from the code) don't really log transactions and they only get INFORMs no PROPOSALs. we can start writing the transactions after the newleader message and i looked at doing that. the code just got a little to complicated for very little benefit. all the proposals up to the uptodate are already committed, so we aren't waiting on an ack. and we would have to move to writing out the pending proposals until after the newleader or uptodate (for backwards compatibility) and then check if have already started writing in the PROPOSAL handling. it just seemed simpler to wait until the end and do it all at once.
            breed Benjamin Reed added a comment -

            henry could you check the observer handling?

            breed Benjamin Reed added a comment - henry could you check the observer handling?

            Ok, I think this is all fine. I will check this in.

            fournc Camille Fournier added a comment - Ok, I think this is all fine. I will check this in.
            lakshman Laxman added a comment -

            Similar issue occurred in our case as well. The ephemeral exists only in one of the follower even after it's owner session is expired. I wanted to know the exact scenario when this may happen, so that I can add a testcase to verify.

            lakshman Laxman added a comment - Similar issue occurred in our case as well. The ephemeral exists only in one of the follower even after it's owner session is expired. I wanted to know the exact scenario when this may happen, so that I can add a testcase to verify.

            Can someone take a look at this patch? I made it last night to check in to the 3.4 branch but the Zab1_0Test fails when I run it.

            fournc Camille Fournier added a comment - Can someone take a look at this patch? I made it last night to check in to the 3.4 branch but the Zab1_0Test fails when I run it.
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12502472/ZOOKEEPER-1264-branch34.patch
            against trunk revision 1196819.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 6 new or modified tests.

            -1 patch. The patch command could not apply the patch.

            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/775//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502472/ZOOKEEPER-1264-branch34.patch against trunk revision 1196819. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/775//console This message is automatically generated.
            mahadev Mahadev Konar added a comment -

            Camille,
            Are you debugging the test failure in 3.4 or waiting for others to take a look?

            mahadev Mahadev Konar added a comment - Camille, Are you debugging the test failure in 3.4 or waiting for others to take a look?

            I'm looking at it. The test that fails is not a new test, it's testAbandonBeforeACKEpoch. I have no idea why it is failing.

            fournc Camille Fournier added a comment - I'm looking at it. The test that fails is not a new test, it's testAbandonBeforeACKEpoch. I have no idea why it is failing.

            Oh now I see. Because 1192 introduced fixes into leader election that added stuff to the Zab1_0Test that I missed. Why in the world do we have leader election bugs going only into trunk instead of into 3.4 as well??? Not good.

            fournc Camille Fournier added a comment - Oh now I see. Because 1192 introduced fixes into leader election that added stuff to the Zab1_0Test that I missed. Why in the world do we have leader election bugs going only into trunk instead of into 3.4 as well??? Not good.

            I still cannot manage a clean patch of this issue and tests to the 3.4 branch. This is my latest attempt, which still fails the Zab1_0Test. Can someone else please take a look here?

            fournc Camille Fournier added a comment - I still cannot manage a clean patch of this issue and tests to the 3.4 branch. This is my latest attempt, which still fails the Zab1_0Test. Can someone else please take a look here?
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12502600/ZOOKEEPER-1264-34-bad.patch
            against trunk revision 1197891.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 6 new or modified tests.

            -1 patch. The patch command could not apply the patch.

            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/776//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502600/ZOOKEEPER-1264-34-bad.patch against trunk revision 1197891. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/776//console This message is automatically generated.

            I'll have a look.

            fpj Flavio Paiva Junqueira added a comment - I'll have a look.

            This is the patch to trunk, verifying via hudson that it passes everything.

            fournc Camille Fournier added a comment - This is the patch to trunk, verifying via hudson that it passes everything.

            Ben added a check to testNormalRun at line 502:

            Assert.assertEquals(1, l.self.getAcceptedEpoch());

            This line consistently fails for me. I'm not sure why this would be failing in the 3.4 branch but not trunk.

            fournc Camille Fournier added a comment - Ben added a check to testNormalRun at line 502: Assert.assertEquals(1, l.self.getAcceptedEpoch()); This line consistently fails for me. I'm not sure why this would be failing in the 3.4 branch but not trunk.

            I know where the problem is. Ben assumes that acceptedEpoch is set atomically before the leader replies to the client with LEADERINFO, but it doesn't. The leader sets it after returning from getEpochToPropose. This is a bug. I'll upload a small patch in a sec

            shralex Alexander Shraer added a comment - I know where the problem is. Ben assumes that acceptedEpoch is set atomically before the leader replies to the client with LEADERINFO, but it doesn't. The leader sets it after returning from getEpochToPropose. This is a bug. I'll upload a small patch in a sec
            hadoopqa Hadoop QA added a comment -

            +1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12502605/ZOOKEEPER-1264-latest.patch
            against trunk revision 1197891.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 6 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/777//testReport/
            Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/777//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/777//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502605/ZOOKEEPER-1264-latest.patch against trunk revision 1197891. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/777//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/777//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/777//console This message is automatically generated.
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12502606/ZOOKEEPER-ACCEPTEDEPOCH.patch
            against trunk revision 1197891.

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            -1 patch. The patch command could not apply the patch.

            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/778//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502606/ZOOKEEPER-ACCEPTEDEPOCH.patch against trunk revision 1197891. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/778//console This message is automatically generated.

            +1, looks right to me.

            fpj Flavio Paiva Junqueira added a comment - +1, looks right to me.

            Final version of fix for 3.4 branch

            fournc Camille Fournier added a comment - Final version of fix for 3.4 branch
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12502608/ZOOKEEPER-1264-br34-final.patch
            against trunk revision 1197891.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 6 new or modified tests.

            -1 patch. The patch command could not apply the patch.

            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/779//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502608/ZOOKEEPER-1264-br34-final.patch against trunk revision 1197891. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/779//console This message is automatically generated.

            final trunk patch

            fournc Camille Fournier added a comment - final trunk patch
            hadoopqa Hadoop QA added a comment -

            +1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12502617/ZOOKEEPER-1264-final.patch
            against trunk revision 1197891.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 6 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/781//testReport/
            Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/781//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/781//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502617/ZOOKEEPER-1264-final.patch against trunk revision 1197891. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/781//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/781//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/781//console This message is automatically generated.
            hadoopqa Hadoop QA added a comment -

            +1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12502617/ZOOKEEPER-1264-final.patch
            against trunk revision 1197891.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 6 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/782//testReport/
            Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/782//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/782//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502617/ZOOKEEPER-1264-final.patch against trunk revision 1197891. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/782//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/782//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/782//console This message is automatically generated.
            hudson Hudson added a comment -

            Integrated in ZooKeeper-trunk #1357 (See https://builds.apache.org/job/ZooKeeper-trunk/1357/)
            ZOOKEEPER-1264. FollowerResyncConcurrencyTest failing intermittently.
            ZOOKEEPER-1282. Learner.java not following Zab 1.0 protocol - setCurrentEpoch should be done upon receipt of NEWLEADER (before acking it) and not upon receipt of UPTODATE.
            ZOOKEEPER-1291. AcceptedEpoch not updated at leader before it proposes the epoch to followers.

            camille : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1198053
            Files :

            • /zookeeper/trunk/CHANGES.txt
            • /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/Leader.java
            • /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/Learner.java
            • /zookeeper/trunk/src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java
            • /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/FollowerResyncConcurrencyTest.java
            hudson Hudson added a comment - Integrated in ZooKeeper-trunk #1357 (See https://builds.apache.org/job/ZooKeeper-trunk/1357/ ) ZOOKEEPER-1264 . FollowerResyncConcurrencyTest failing intermittently. ZOOKEEPER-1282 . Learner.java not following Zab 1.0 protocol - setCurrentEpoch should be done upon receipt of NEWLEADER (before acking it) and not upon receipt of UPTODATE. ZOOKEEPER-1291 . AcceptedEpoch not updated at leader before it proposes the epoch to followers. camille : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1198053 Files : /zookeeper/trunk/CHANGES.txt /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/Leader.java /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/Learner.java /zookeeper/trunk/src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/FollowerResyncConcurrencyTest.java

            People

              fournc Camille Fournier
              phunt Patrick D. Hunt
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: