ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-508

proposals and commits for DIFF and Truncate messages from the leader to followers is buggy.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.1, 3.3.0
    • Component/s: quorum
    • Labels:
      None

      Description

      The proposals and commits sent by the leader after it asks the followers to truncate there logs or starts sending a diff has missing messages which causes out of order commits messages and causes the followers to shutdown because of these out of order commits.

      1. ZOOKEEPER-508.patch
        22 kB
        Mahadev konar
      2. ZOOKEEPER-508.patch-3.2
        22 kB
        Mahadev konar
      3. ZOOKEEPER-508.patch
        22 kB
        Mahadev konar
      4. ZOOKEEPER-508.patch
        21 kB
        Mahadev konar
      5. ZOOKEEPER-508.patch
        20 kB
        Benjamin Reed
      6. ZOOKEEPER-508.patch
        15 kB
        Mahadev konar

        Issue Links

          Activity

          Mahadev konar created issue -
          Mahadev konar made changes -
          Field Original Value New Value
          Link This issue is related to ZOOKEEPER-483 [ ZOOKEEPER-483 ]
          Mahadev konar made changes -
          Component/s quorum [ 12312379 ]
          Hide
          Mahadev konar added a comment -

          this fixes the issue so that we sent the right packets when a follower syncs with the leader. also includes ZOOKEEPER-483.

          Show
          Mahadev konar added a comment - this fixes the issue so that we sent the right packets when a follower syncs with the leader. also includes ZOOKEEPER-483 .
          Mahadev konar made changes -
          Attachment ZOOKEEPER-508.patch [ 12416937 ]
          Hide
          Benjamin Reed added a comment -

          added a testcase for the DIFF problem. still not fixed.

          Show
          Benjamin Reed added a comment - added a testcase for the DIFF problem. still not fixed.
          Benjamin Reed made changes -
          Attachment ZOOKEEPER-508.patch [ 12417095 ]
          Hide
          Mahadev konar added a comment -

          i fugured out the reason why it fails on the assertion errors != 0. The whole scenario of 483 fialing with truncate is this

          • the test case shutdowns all the followers
          • the leader does not realize that its lost the leadership, becasue the time we ping to see if the leader is still the leader (maybe 1 sec) is greater than the time hte followers actually take to shutdown and get back and in sync with the leader
          • so the leader never shutsdown any of its stuff (no NIO rejection or nething else)

          so in your case, sometimes the client conencts to the leader and will never see errors. On the other hand sometimes it may pas s on connection to other followers and your testcase will pass. So we cannot really say that your test case is fool proof.

          Show
          Mahadev konar added a comment - i fugured out the reason why it fails on the assertion errors != 0. The whole scenario of 483 fialing with truncate is this the test case shutdowns all the followers the leader does not realize that its lost the leadership, becasue the time we ping to see if the leader is still the leader (maybe 1 sec) is greater than the time hte followers actually take to shutdown and get back and in sync with the leader so the leader never shutsdown any of its stuff (no NIO rejection or nething else) so in your case, sometimes the client conencts to the leader and will never see errors. On the other hand sometimes it may pas s on connection to other followers and your testcase will pass. So we cannot really say that your test case is fool proof.
          Hide
          Mahadev konar added a comment -

          this patch includes fix for ZOOKEEPER-508, ZOOKEEPER-509, ZOOKEEPER-483. It also includes the test cases for each of them.

          Show
          Mahadev konar added a comment - this patch includes fix for ZOOKEEPER-508 , ZOOKEEPER-509 , ZOOKEEPER-483 . It also includes the test cases for each of them.
          Mahadev konar made changes -
          Attachment ZOOKEEPER-508.patch [ 12417189 ]
          Mahadev konar made changes -
          Link This issue is related to ZOOKEEPER-483 [ ZOOKEEPER-483 ]
          Mahadev konar made changes -
          Link This issue incorporates ZOOKEEPER-483 [ ZOOKEEPER-483 ]
          Mahadev konar made changes -
          Link This issue blocks ZOOKEEPER-509 [ ZOOKEEPER-509 ]
          Mahadev konar made changes -
          Link This issue blocks ZOOKEEPER-483 [ ZOOKEEPER-483 ]
          Mahadev konar made changes -
          Link This issue blocks ZOOKEEPER-509 [ ZOOKEEPER-509 ]
          Mahadev konar made changes -
          Link This issue blocks ZOOKEEPER-483 [ ZOOKEEPER-483 ]
          Mahadev konar made changes -
          Link This issue incorporates ZOOKEEPER-509 [ ZOOKEEPER-509 ]
          Mahadev konar made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Mahadev konar added a comment -

          I have a patch for the 3.2 branch, will upload is as soon as hudson is done running this patch.

          Show
          Mahadev konar added a comment - I have a patch for the 3.2 branch, will upload is as soon as hudson is done running this patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12417189/ZOOKEEPER-508.patch
          against trunk revision 803300.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          -1 release audit. The applied patch generated 178 release audit warnings (more than the trunk's current 177 warnings).

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/188/testReport/
          Release audit warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/188/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/188/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/188/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417189/ZOOKEEPER-508.patch against trunk revision 803300. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 178 release audit warnings (more than the trunk's current 177 warnings). -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/188/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/188/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/188/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/188/console This message is automatically generated.
          Hide
          Mahadev konar added a comment -

          attached missing header to the file.

          Show
          Mahadev konar added a comment - attached missing header to the file.
          Mahadev konar made changes -
          Attachment ZOOKEEPER-508.patch [ 12417207 ]
          Mahadev konar made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Mahadev konar made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12417207/ZOOKEEPER-508.patch
          against trunk revision 803300.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/189/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/189/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/189/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417207/ZOOKEEPER-508.patch against trunk revision 803300. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/189/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/189/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/189/console This message is automatically generated.
          Mahadev konar made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Mahadev konar added a comment -

          patch for 3.2

          fixes all the issues, also the new one found by hudson running the tests.

          Show
          Mahadev konar added a comment - patch for 3.2 fixes all the issues, also the new one found by hudson running the tests.
          Mahadev konar made changes -
          Attachment ZOOKEEPER-508.patch-3.2 [ 12417338 ]
          Hide
          Mahadev konar added a comment -

          patch for trunk.

          • fixes all the issues with tests.
          Show
          Mahadev konar added a comment - patch for trunk. fixes all the issues with tests.
          Mahadev konar made changes -
          Attachment ZOOKEEPER-508.patch [ 12417339 ]
          Mahadev konar made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12417339/ZOOKEEPER-508.patch
          against trunk revision 803300.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/190/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/190/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/190/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417339/ZOOKEEPER-508.patch against trunk revision 803300. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/190/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/190/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/190/console This message is automatically generated.
          Hide
          Benjamin Reed added a comment -

          +1 looks good. simple fix!

          Show
          Benjamin Reed added a comment - +1 looks good. simple fix!
          Hide
          Mahadev konar added a comment -

          I just committed this.

          Show
          Mahadev konar added a comment - I just committed this.
          Mahadev konar made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Release Note includes the fix for ZOOKEEPER-483, ZOOKEEPER-509.
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in ZooKeeper-trunk #426 (See http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/426/)
          . proposals and commits for DIFF and Truncate messages from the leader to the followers is buggy. (mahadev and ben via mahadev)

          Show
          Hudson added a comment - Integrated in ZooKeeper-trunk #426 (See http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/426/ ) . proposals and commits for DIFF and Truncate messages from the leader to the followers is buggy. (mahadev and ben via mahadev)
          Patrick Hunt made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          1d 1m 2 Mahadev konar 21/Aug/09 23:47
          Open Open Patch Available Patch Available
          3d 4h 17m 3 Mahadev konar 21/Aug/09 23:57
          Patch Available Patch Available Resolved Resolved
          3d 6h 59m 1 Mahadev konar 25/Aug/09 06:56
          Resolved Resolved Closed Closed
          11d 16h 39m 1 Patrick Hunt 05/Sep/09 23:36

            People

            • Assignee:
              Mahadev konar
              Reporter:
              Mahadev konar
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development