Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.4.3
    • Fix Version/s: 3.6.0
    • Component/s: server
    • Labels:
      None

      Description

      This feature reused some of the mechanism already provided by ReadOnlyZooKeeper (ZOOKEEPER-704) but implemented in a different way

      Goal: read-only clients should be able to connect to the observer or continue to read data from the observer event when there is an outage of underling quorum. This means that it is possible for the observer to provide 100% read uptime for read-only local session (ZOOKEEPER-1147)

      Implementation:
      The observer don't tear down itself when it lose connection with the leader. It only close the connection associated with non read-only sessions and global sessions. So the client can try other observer if this is a temporal failure.

      During the outage, the observer switch to read-only mode. All the pending and future write requests get will get NOT_READONLY error. Read-only state transition is sent to all session on that observer. The observer only accepts a new connection from a read-only client.

      When the observer is able to reconnect to the leader. It sends state transition (CONNECTED_STATE) to all current session. If it is able to synchronize with the leader using DIFF, the steam of txns is sent through the commit processor instead of applying to the DataTree directly to prevent raise condition between in-flight read requests (see ZOOKEEPER-1505). The client will receive watch events correctly and can start issuing write requests.

      However, if the observer is getting the snapshot. It need to drop all the connection since it cannot fire a watch correctly.

      1. ZOOKEEPER-1607.patch
        47 kB
        Raul Gutierrez Segales

        Issue Links

          Activity

          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12609067/ZOOKEEPER-1607.patch
          against trunk revision 1586200.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2031//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12609067/ZOOKEEPER-1607.patch against trunk revision 1586200. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2031//console This message is automatically generated.
          Michi Mutsuzaki made changes -
          Fix Version/s 3.6.0 [ 12326518 ]
          Fix Version/s 3.5.0 [ 12316644 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12609067/ZOOKEEPER-1607.patch
          against trunk revision 1533161.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1706//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1706//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1706//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12609067/ZOOKEEPER-1607.patch against trunk revision 1533161. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1706//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1706//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1706//console This message is automatically generated.
          Raul Gutierrez Segales made changes -
          Attachment ZOOKEEPER-1607.patch [ 12609067 ]
          Hide
          Raul Gutierrez Segales added a comment -

          The prev patch had remaining bits and pieces of an internal patch to keep stats using Twitter's stats-util - soz.

          Show
          Raul Gutierrez Segales added a comment - The prev patch had remaining bits and pieces of an internal patch to keep stats using Twitter's stats-util - soz.
          Raul Gutierrez Segales made changes -
          Attachment ZOOKEEPER-1607.patch [ 12609061 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12609061/ZOOKEEPER-1607.patch
          against trunk revision 1533161.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1705//testReport/
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1705//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12609061/ZOOKEEPER-1607.patch against trunk revision 1533161. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1705//testReport/ Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1705//console This message is automatically generated.
          Raul Gutierrez Segales made changes -
          Attachment ZOOKEEPER-1607.patch [ 12609061 ]
          Hide
          Raul Gutierrez Segales added a comment -

          Chatted with Thawan about this and this still probably has to change but I wanted to go ahead and post this for any interested passerby (since local sessions support has been merged).

          Show
          Raul Gutierrez Segales added a comment - Chatted with Thawan about this and this still probably has to change but I wanted to go ahead and post this for any interested passerby (since local sessions support has been merged).
          Raul Gutierrez Segales made changes -
          Attachment persistent-read-only-for-observers.patch [ 12603061 ]
          Patrick Hunt made changes -
          Fix Version/s 3.5.0 [ 12316644 ]
          Patrick Hunt made changes -
          Assignee Raul Gutierrez Segales [ rgs ]
          Hide
          Raul Gutierrez Segales added a comment -

          Arrrg I guess dependent patches aren't applied

          Show
          Raul Gutierrez Segales added a comment - Arrrg I guess dependent patches aren't applied
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12603061/persistent-read-only-for-observers.patch
          against trunk revision 1522079.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1578//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12603061/persistent-read-only-for-observers.patch against trunk revision 1522079. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1578//console This message is automatically generated.
          Raul Gutierrez Segales made changes -
          Attachment persistent-read-only-for-observers.patch [ 12603061 ]
          Hide
          Raul Gutierrez Segales added a comment -

          New version of the patch with tests. Also - this is generated with git diff -p so it should be Hadoop QA friendly.

          Show
          Raul Gutierrez Segales added a comment - New version of the patch with tests. Also - this is generated with git diff -p so it should be Hadoop QA friendly.
          Raul Gutierrez Segales made changes -
          Attachment 0001-RFC-Don-t-tear-down-an-Observer-when-we-lose-connect.patch [ 12597187 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12597187/0001-RFC-Don-t-tear-down-an-Observer-when-we-lose-connect.patch
          against trunk revision 1503101.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1532//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12597187/0001-RFC-Don-t-tear-down-an-Observer-when-we-lose-connect.patch against trunk revision 1503101. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1532//console This message is automatically generated.
          Raul Gutierrez Segales made changes -
          Attachment 0001-RFC-Don-t-tear-down-an-Observer-when-we-lose-connect.patch [ 12597187 ]
          Raul Gutierrez Segales made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Raul Gutierrez Segales added a comment -

          Here goes an RFC diff.

          With this change an Observer will do the following when it loses connection to the leader:

          a) it'll close the datatree (i.e.: the txnlog)
          b) it'll signal a WatchedEvent to all clients informing a new of a
          new KeeperState (Connected.ReadOnly)
          c) any writes will now fail with not-read-only-operation
          d) it'll start looking for a leader from the QuorumPeer thread (the
          main thread) without shutting down the ObserverZooKeeperServer instance
          so clients are preserved and can keep reading (watches will have to
          be set again when we transition back to SyncConnected though)
          e) when a leader is found (again) it'll instantiate a new ZKDatabase
          and sync up with the leader using that
          f) once the new ZKDatabase is up to date it'll use that to replace the
          old/current ZkDatabase from ObserverZooKeeperServer
          g) it'll signal a WatchedEvent to all clients (new KeeperState is
          SyncConnected)
          h) writes can now happen (and watches need to be set again)

          Thoughts?

          Show
          Raul Gutierrez Segales added a comment - Here goes an RFC diff. With this change an Observer will do the following when it loses connection to the leader: a) it'll close the datatree (i.e.: the txnlog) b) it'll signal a WatchedEvent to all clients informing a new of a new KeeperState (Connected.ReadOnly) c) any writes will now fail with not-read-only-operation d) it'll start looking for a leader from the QuorumPeer thread (the main thread) without shutting down the ObserverZooKeeperServer instance so clients are preserved and can keep reading (watches will have to be set again when we transition back to SyncConnected though) e) when a leader is found (again) it'll instantiate a new ZKDatabase and sync up with the leader using that f) once the new ZKDatabase is up to date it'll use that to replace the old/current ZkDatabase from ObserverZooKeeperServer g) it'll signal a WatchedEvent to all clients (new KeeperState is SyncConnected) h) writes can now happen (and watches need to be set again) Thoughts?
          Thawan Kooburat made changes -
          Field Original Value New Value
          Link This issue depends upon ZOOKEEPER-1147 [ ZOOKEEPER-1147 ]
          Thawan Kooburat created issue -

            People

            • Assignee:
              Raul Gutierrez Segales
              Reporter:
              Thawan Kooburat
            • Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:

                Development