Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-2678

Large databases take a long time to regain a quorum

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.4.9, 3.5.2
    • Fix Version/s: 3.4.10, 3.5.3, 3.6.0
    • Component/s: server
    • Labels:
      None

      Description

      I know this is long but please here me out.

      I recently inherited a massive zookeeper ensemble. The snapshot is 3.4 GB on disk. Because of its massive size we have been running into a number of issues. There are lots of problems that we hope to fix with tuning GC etc, but the big one right now that is blocking us making a lot of progress on the rest of them is that when we lose a quorum because the leader left, for what ever reason, it can take well over 5 mins for a new quorum to be established. So we cannot tune the leader without risking downtime.

      We traced down where the time was being spent and found that each server was clearing the database so it would be read back in again before leader election even started. Then as part of the sync phase each server will write out a snapshot to checkpoint the progress it made as part of the sync.

      I will be putting up a patch shortly with some proposed changes in it.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user revans2 opened a pull request:

          https://github.com/apache/zookeeper/pull/157

          ZOOKEEPER-2678: Discovery and Sync can take a very long time on large DB

          This patch addresses recovery time when a leader is lost on a large DB.

          It does this by not clearing the DB before leader election begins, and by avoiding taking a snapshot as part of the SYNC phase, specifically for a DIFF sync. It does this by buffering the proposals and commits just like the code currently does for proposals/commits sent after the NEWLEADER and before the UPTODATE messages.

          If a SNAP is sent we cannot avoid writing out the full snapshot because there is no other way to make sure the disk DB is in sync with what is in memory. So any edits to the edit log before a background snapshot happened could possibly be applied on top of an incorrect snapshot.

          This same optimization should work for TRUNC too, but I opted not to do it for TRUNC because TRUNC is rare and TRUNC by its very nature already forces the DB to be reread after the edit logs are modified. So it would still not be fast.

          In practice this makes it so instead of taking 5+ mins for the cluster to recover from losing a leader it now takes about 3 seconds.

          I am happy to port this to 3.5. if it looks good.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/revans2/zookeeper ZOOKEEPER-2678

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/zookeeper/pull/157.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #157


          commit 5aa25620e0189b28d7040305272be2fda28126fb
          Author: Robert (Bobby) Evans <evans@yahoo-inc.com>
          Date: 2017-01-19T19:50:32Z

          ZOOKEEPER-2678: Discovery and Sync can take a very long time on large DBs


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user revans2 opened a pull request: https://github.com/apache/zookeeper/pull/157 ZOOKEEPER-2678 : Discovery and Sync can take a very long time on large DB This patch addresses recovery time when a leader is lost on a large DB. It does this by not clearing the DB before leader election begins, and by avoiding taking a snapshot as part of the SYNC phase, specifically for a DIFF sync. It does this by buffering the proposals and commits just like the code currently does for proposals/commits sent after the NEWLEADER and before the UPTODATE messages. If a SNAP is sent we cannot avoid writing out the full snapshot because there is no other way to make sure the disk DB is in sync with what is in memory. So any edits to the edit log before a background snapshot happened could possibly be applied on top of an incorrect snapshot. This same optimization should work for TRUNC too, but I opted not to do it for TRUNC because TRUNC is rare and TRUNC by its very nature already forces the DB to be reread after the edit logs are modified. So it would still not be fast. In practice this makes it so instead of taking 5+ mins for the cluster to recover from losing a leader it now takes about 3 seconds. I am happy to port this to 3.5. if it looks good. You can merge this pull request into a Git repository by running: $ git pull https://github.com/revans2/zookeeper ZOOKEEPER-2678 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/157.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #157 commit 5aa25620e0189b28d7040305272be2fda28126fb Author: Robert (Bobby) Evans <evans@yahoo-inc.com> Date: 2017-01-19T19:50:32Z ZOOKEEPER-2678 : Discovery and Sync can take a very long time on large DBs
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fpj commented on a diff in the pull request:

          https://github.com/apache/zookeeper/pull/157#discussion_r98336253

          — Diff: src/java/main/org/apache/zookeeper/server/quorum/Learner.java —
          @@ -364,10 +367,12 @@ else if (qp.getType() == Leader.SNAP) {

          long lastQueued = 0;

          • // in V1.0 we take a snapshot when we get the NEWLEADER message, but in pre V1.0
            + // in Zab V1.0 (ZK 3.4+) we might take a snapshot when we get the NEWLEADER message, but in pre V1.0
            // we take the snapshot at the UPDATE, since V1.0 also gets the UPDATE (after the NEWLEADER)
            // we need to make sure that we don't take the snapshot twice.
          • boolean snapshotTaken = false;
            + boolean isPreZAB1_0 = true;
            + //If we are not going to take the snapshot be sure the edits are not applied in memory
            + boolean writeToEditLog = !snapshotNeeded;
              • End diff –

          The changes here are using `edit` to refer to `txns`. I'd rather use `txn` to be consistent across the project. Specifically here, you're using `EditLog` to refer to the `TxnLog`, please change accordingly to have it consistent across the project.

          Show
          githubbot ASF GitHub Bot added a comment - Github user fpj commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/157#discussion_r98336253 — Diff: src/java/main/org/apache/zookeeper/server/quorum/Learner.java — @@ -364,10 +367,12 @@ else if (qp.getType() == Leader.SNAP) { long lastQueued = 0; // in V1.0 we take a snapshot when we get the NEWLEADER message, but in pre V1.0 + // in Zab V1.0 (ZK 3.4+) we might take a snapshot when we get the NEWLEADER message, but in pre V1.0 // we take the snapshot at the UPDATE, since V1.0 also gets the UPDATE (after the NEWLEADER) // we need to make sure that we don't take the snapshot twice. boolean snapshotTaken = false; + boolean isPreZAB1_0 = true; + //If we are not going to take the snapshot be sure the edits are not applied in memory + boolean writeToEditLog = !snapshotNeeded; End diff – The changes here are using `edit` to refer to `txns`. I'd rather use `txn` to be consistent across the project. Specifically here, you're using `EditLog` to refer to the `TxnLog`, please change accordingly to have it consistent across the project.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fpj commented on a diff in the pull request:

          https://github.com/apache/zookeeper/pull/157#discussion_r98337403

          — Diff: src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java —
          @@ -839,6 +839,13 @@ public void converseWithFollower(InputArchive ia, OutputArchive oa,
          Assert.assertEquals(1, f.self.getAcceptedEpoch());
          Assert.assertEquals(1, f.self.getCurrentEpoch());

          + //Wait for the edits to be written out
          — End diff –

          I need to think some more whether it makes any sense to add test cases for this. The test cases we already have probably cover this enough given that there is no real change of behavior.

          This change here is necessary, though. We don't really care about time in general in our tests because we can never be sure of the timing we will get across runs and with different settings.

          Show
          githubbot ASF GitHub Bot added a comment - Github user fpj commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/157#discussion_r98337403 — Diff: src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java — @@ -839,6 +839,13 @@ public void converseWithFollower(InputArchive ia, OutputArchive oa, Assert.assertEquals(1, f.self.getAcceptedEpoch()); Assert.assertEquals(1, f.self.getCurrentEpoch()); + //Wait for the edits to be written out — End diff – I need to think some more whether it makes any sense to add test cases for this. The test cases we already have probably cover this enough given that there is no real change of behavior. This change here is necessary, though. We don't really care about time in general in our tests because we can never be sure of the timing we will get across runs and with different settings.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user fpj commented on the issue:

          https://github.com/apache/zookeeper/pull/157

          Thanks for the patch @revans2 . It makes sense to port this change to both 3.5 and master.

          Show
          githubbot ASF GitHub Bot added a comment - Github user fpj commented on the issue: https://github.com/apache/zookeeper/pull/157 Thanks for the patch @revans2 . It makes sense to port this change to both 3.5 and master.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user revans2 commented on a diff in the pull request:

          https://github.com/apache/zookeeper/pull/157#discussion_r98449817

          — Diff: src/java/main/org/apache/zookeeper/server/quorum/Learner.java —
          @@ -364,10 +367,12 @@ else if (qp.getType() == Leader.SNAP) {

          long lastQueued = 0;

          • // in V1.0 we take a snapshot when we get the NEWLEADER message, but in pre V1.0
            + // in Zab V1.0 (ZK 3.4+) we might take a snapshot when we get the NEWLEADER message, but in pre V1.0
            // we take the snapshot at the UPDATE, since V1.0 also gets the UPDATE (after the NEWLEADER)
            // we need to make sure that we don't take the snapshot twice.
          • boolean snapshotTaken = false;
            + boolean isPreZAB1_0 = true;
            + //If we are not going to take the snapshot be sure the edits are not applied in memory
            + boolean writeToEditLog = !snapshotNeeded;
              • End diff –

          Sorry about that I am still a bit new at the internal terminology. I will update it.

          Show
          githubbot ASF GitHub Bot added a comment - Github user revans2 commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/157#discussion_r98449817 — Diff: src/java/main/org/apache/zookeeper/server/quorum/Learner.java — @@ -364,10 +367,12 @@ else if (qp.getType() == Leader.SNAP) { long lastQueued = 0; // in V1.0 we take a snapshot when we get the NEWLEADER message, but in pre V1.0 + // in Zab V1.0 (ZK 3.4+) we might take a snapshot when we get the NEWLEADER message, but in pre V1.0 // we take the snapshot at the UPDATE, since V1.0 also gets the UPDATE (after the NEWLEADER) // we need to make sure that we don't take the snapshot twice. boolean snapshotTaken = false; + boolean isPreZAB1_0 = true; + //If we are not going to take the snapshot be sure the edits are not applied in memory + boolean writeToEditLog = !snapshotNeeded; End diff – Sorry about that I am still a bit new at the internal terminology. I will update it.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user revans2 commented on the issue:

          https://github.com/apache/zookeeper/pull/157

          @fpj Thanks for the review I will update the comments and start porting it to other lines.

          Show
          githubbot ASF GitHub Bot added a comment - Github user revans2 commented on the issue: https://github.com/apache/zookeeper/pull/157 @fpj Thanks for the review I will update the comments and start porting it to other lines.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user revans2 opened a pull request:

          https://github.com/apache/zookeeper/pull/158

          ZOOKEEPER-2678: Discovery and Sync can take a very long time on large DBs

          This is the 3.5 version of #157

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/revans2/zookeeper ZOOKEEPER-2678-3.5

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/zookeeper/pull/158.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #158


          commit f871601abaf2420c674bb78658ceeb5083b3b04f
          Author: Robert (Bobby) Evans <evans@yahoo-inc.com>
          Date: 2017-01-19T19:50:32Z

          ZOOKEEPER-2678: Discovery and Sync can take a very long time on large DBs


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user revans2 opened a pull request: https://github.com/apache/zookeeper/pull/158 ZOOKEEPER-2678 : Discovery and Sync can take a very long time on large DBs This is the 3.5 version of #157 You can merge this pull request into a Git repository by running: $ git pull https://github.com/revans2/zookeeper ZOOKEEPER-2678 -3.5 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/158.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #158 commit f871601abaf2420c674bb78658ceeb5083b3b04f Author: Robert (Bobby) Evans <evans@yahoo-inc.com> Date: 2017-01-19T19:50:32Z ZOOKEEPER-2678 : Discovery and Sync can take a very long time on large DBs
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user revans2 opened a pull request:

          https://github.com/apache/zookeeper/pull/159

          ZOOKEEPER-2678: Discovery and Sync can take a very long time on large DBs

          This is the master version of #157

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/revans2/zookeeper ZOOKEEPER-2678-master

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/zookeeper/pull/159.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #159



          Show
          githubbot ASF GitHub Bot added a comment - GitHub user revans2 opened a pull request: https://github.com/apache/zookeeper/pull/159 ZOOKEEPER-2678 : Discovery and Sync can take a very long time on large DBs This is the master version of #157 You can merge this pull request into a Git repository by running: $ git pull https://github.com/revans2/zookeeper ZOOKEEPER-2678 -master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/159.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #159
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user revans2 commented on the issue:

          https://github.com/apache/zookeeper/pull/157

          @fpj I addressed your review comments, and I also found another race in the ZAB test that I addressed. Apparently the log line I removed was taking enough time for the transactions to be fully flushed. When I removed it the test would occasionally fail.

          Please also take a look at #158 and #159 for master and branch 3.5.

          Show
          githubbot ASF GitHub Bot added a comment - Github user revans2 commented on the issue: https://github.com/apache/zookeeper/pull/157 @fpj I addressed your review comments, and I also found another race in the ZAB test that I addressed. Apparently the log line I removed was taking enough time for the transactions to be fully flushed. When I removed it the test would occasionally fail. Please also take a look at #158 and #159 for master and branch 3.5.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. GitHub Pull Request Build

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/265//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/265//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/265//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/265//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/265//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/265//console This message is automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user eribeiro commented on the issue:

          https://github.com/apache/zookeeper/pull/157

          Hey @revans2, FYI. I was able to apply the both #158 and #159 without any explicit conflict on `branch-3.5` and `master` and #157 on branch-3.4 (but not on the others cited previously). So... *you really don't need 3 PR, just 2: one for branch-3.4 and other for `branch-3.5` and `master`*, right?

          If I am right then putting a comment to apply #158 to both branch-3.4 and master should be enough, *IMHO*.

          Show
          githubbot ASF GitHub Bot added a comment - Github user eribeiro commented on the issue: https://github.com/apache/zookeeper/pull/157 Hey @revans2, FYI. I was able to apply the both #158 and #159 without any explicit conflict on `branch-3.5` and `master` and #157 on branch-3.4 (but not on the others cited previously). So... * you really don't need 3 PR, just 2: one for branch-3.4 and other for `branch-3.5` and `master` *, right? If I am right then putting a comment to apply #158 to both branch-3.4 and master should be enough, * IMHO *.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user revans2 commented on the issue:

          https://github.com/apache/zookeeper/pull/157

          @eribeiro will do. It is a clean cherry pick between master and 3.5

          Show
          githubbot ASF GitHub Bot added a comment - Github user revans2 commented on the issue: https://github.com/apache/zookeeper/pull/157 @eribeiro will do. It is a clean cherry pick between master and 3.5
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user revans2 commented on the issue:

          https://github.com/apache/zookeeper/pull/158

          Closing this because master applies cleanly to 3.5 as well

          Show
          githubbot ASF GitHub Bot added a comment - Github user revans2 commented on the issue: https://github.com/apache/zookeeper/pull/158 Closing this because master applies cleanly to 3.5 as well
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user revans2 closed the pull request at:

          https://github.com/apache/zookeeper/pull/158

          Show
          githubbot ASF GitHub Bot added a comment - Github user revans2 closed the pull request at: https://github.com/apache/zookeeper/pull/158
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user revans2 commented on the issue:

          https://github.com/apache/zookeeper/pull/159

          This also applies cleanly to the 3.5 line #157 is for the 3.4 line

          Show
          githubbot ASF GitHub Bot added a comment - Github user revans2 commented on the issue: https://github.com/apache/zookeeper/pull/159 This also applies cleanly to the 3.5 line #157 is for the 3.4 line
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user afine commented on a diff in the pull request:

          https://github.com/apache/zookeeper/pull/157#discussion_r98574533

          — Diff: src/java/main/org/apache/zookeeper/server/quorum/Learner.java —
          @@ -321,13 +321,16 @@ protected void syncWithLeader(long newLeaderZxid) throws IOException, Interrupte
          QuorumPacket ack = new QuorumPacket(Leader.ACK, 0, null, null);
          QuorumPacket qp = new QuorumPacket();
          long newEpoch = ZxidUtils.getEpochFromZxid(newLeaderZxid);

          • readPacket(qp);
            + //In the DIFF case we don't need to do a snapshot because the transactions will sync on top of any existing snapshot
              • End diff –

          nit: I think we generally put spaces after the //

          Show
          githubbot ASF GitHub Bot added a comment - Github user afine commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/157#discussion_r98574533 — Diff: src/java/main/org/apache/zookeeper/server/quorum/Learner.java — @@ -321,13 +321,16 @@ protected void syncWithLeader(long newLeaderZxid) throws IOException, Interrupte QuorumPacket ack = new QuorumPacket(Leader.ACK, 0, null, null); QuorumPacket qp = new QuorumPacket(); long newEpoch = ZxidUtils.getEpochFromZxid(newLeaderZxid); readPacket(qp); + //In the DIFF case we don't need to do a snapshot because the transactions will sync on top of any existing snapshot End diff – nit: I think we generally put spaces after the //
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user afine commented on a diff in the pull request:

          https://github.com/apache/zookeeper/pull/157#discussion_r98574493

          — Diff: src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java —
          @@ -839,12 +839,25 @@ public void converseWithFollower(InputArchive ia, OutputArchive oa,
          Assert.assertEquals(1, f.self.getAcceptedEpoch());
          Assert.assertEquals(1, f.self.getCurrentEpoch());

          + //Wait for the transactions to be written out. The thread that writes them out
          + // does not send anything back when it is done.
          + long start = System.currentTimeMillis();
          + while (createSessionZxid != f.fzk.getLastProcessedZxid() && (System.currentTimeMillis() - start) < 50) {
          — End diff –

          just an idea, not sure if it is worth the effort and it may be outside the scope of this patch.

          we could play with the test infrastructure here a little bit and do some dependency injection in `createFollower` that can let us track if db clearing and snapshotting occurs when expected.

          this may help prevent hard to track race conditions with tests like this in the future.

          Show
          githubbot ASF GitHub Bot added a comment - Github user afine commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/157#discussion_r98574493 — Diff: src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java — @@ -839,12 +839,25 @@ public void converseWithFollower(InputArchive ia, OutputArchive oa, Assert.assertEquals(1, f.self.getAcceptedEpoch()); Assert.assertEquals(1, f.self.getCurrentEpoch()); + //Wait for the transactions to be written out. The thread that writes them out + // does not send anything back when it is done. + long start = System.currentTimeMillis(); + while (createSessionZxid != f.fzk.getLastProcessedZxid() && (System.currentTimeMillis() - start) < 50) { — End diff – just an idea, not sure if it is worth the effort and it may be outside the scope of this patch. we could play with the test infrastructure here a little bit and do some dependency injection in `createFollower` that can let us track if db clearing and snapshotting occurs when expected. this may help prevent hard to track race conditions with tests like this in the future.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hanm commented on a diff in the pull request:

          https://github.com/apache/zookeeper/pull/157#discussion_r98734336

          — Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java —
          @@ -507,9 +507,12 @@ public synchronized void shutdown() {
          if (firstProcessor != null)

          { firstProcessor.shutdown(); }
          • if (zkDb != null) { - zkDb.clear(); - }

            +
            + // There is no need to clear the database
            + // * When a new quorum is established we can still apply the diff
            + // on top of the same zkDb data
            + // * If we fetch a new snapshot from leader, the zkDb will be
            + // cleared anyway before loading the snapshot

          — End diff –

          There is one case we may still want to clear db here - when one of the ZooKeeper critical threads (such as * processors, session trackers) fail, ZooKeeper server will shutdown (see runFromConfig) and consequently invoke ZooKeeper#shutdown. In this case, I don't see a particular reason not to clear the db, though not doing it might be fine (as one could argue the server will be dead anyway), but I tend to lean towards the safe side on cleaning the db. One way to conditionally do that is to add a Boolean parameter to ZooKeeper#shutdown so we can have fine grained control over when to clear db in what code path.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/157#discussion_r98734336 — Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java — @@ -507,9 +507,12 @@ public synchronized void shutdown() { if (firstProcessor != null) { firstProcessor.shutdown(); } if (zkDb != null) { - zkDb.clear(); - } + + // There is no need to clear the database + // * When a new quorum is established we can still apply the diff + // on top of the same zkDb data + // * If we fetch a new snapshot from leader, the zkDb will be + // cleared anyway before loading the snapshot — End diff – There is one case we may still want to clear db here - when one of the ZooKeeper critical threads (such as * processors, session trackers) fail, ZooKeeper server will shutdown (see runFromConfig) and consequently invoke ZooKeeper#shutdown. In this case, I don't see a particular reason not to clear the db, though not doing it might be fine (as one could argue the server will be dead anyway), but I tend to lean towards the safe side on cleaning the db. One way to conditionally do that is to add a Boolean parameter to ZooKeeper#shutdown so we can have fine grained control over when to clear db in what code path.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user revans2 commented on a diff in the pull request:

          https://github.com/apache/zookeeper/pull/157#discussion_r98744573

          — Diff: src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java —
          @@ -839,12 +839,25 @@ public void converseWithFollower(InputArchive ia, OutputArchive oa,
          Assert.assertEquals(1, f.self.getAcceptedEpoch());
          Assert.assertEquals(1, f.self.getCurrentEpoch());

          + //Wait for the transactions to be written out. The thread that writes them out
          + // does not send anything back when it is done.
          + long start = System.currentTimeMillis();
          + while (createSessionZxid != f.fzk.getLastProcessedZxid() && (System.currentTimeMillis() - start) < 50) {
          — End diff –

          It does seem a bit beyond the scope of this. But if you really want me to I can look into it.

          Show
          githubbot ASF GitHub Bot added a comment - Github user revans2 commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/157#discussion_r98744573 — Diff: src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java — @@ -839,12 +839,25 @@ public void converseWithFollower(InputArchive ia, OutputArchive oa, Assert.assertEquals(1, f.self.getAcceptedEpoch()); Assert.assertEquals(1, f.self.getCurrentEpoch()); + //Wait for the transactions to be written out. The thread that writes them out + // does not send anything back when it is done. + long start = System.currentTimeMillis(); + while (createSessionZxid != f.fzk.getLastProcessedZxid() && (System.currentTimeMillis() - start) < 50) { — End diff – It does seem a bit beyond the scope of this. But if you really want me to I can look into it.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user revans2 commented on the issue:

          https://github.com/apache/zookeeper/pull/157

          @hanm I addressed your review comments.

          Show
          githubbot ASF GitHub Bot added a comment - Github user revans2 commented on the issue: https://github.com/apache/zookeeper/pull/157 @hanm I addressed your review comments.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. GitHub Pull Request Build

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/273//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/273//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/273//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/273//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/273//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/273//console This message is automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hanm commented on the issue:

          https://github.com/apache/zookeeper/pull/157

          @revans2 the change looks good, thanks.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/157 @revans2 the change looks good, thanks.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user afine commented on a diff in the pull request:

          https://github.com/apache/zookeeper/pull/157#discussion_r99242422

          — Diff: src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java —
          @@ -839,12 +839,25 @@ public void converseWithFollower(InputArchive ia, OutputArchive oa,
          Assert.assertEquals(1, f.self.getAcceptedEpoch());
          Assert.assertEquals(1, f.self.getCurrentEpoch());

          + //Wait for the transactions to be written out. The thread that writes them out
          + // does not send anything back when it is done.
          + long start = System.currentTimeMillis();
          + while (createSessionZxid != f.fzk.getLastProcessedZxid() && (System.currentTimeMillis() - start) < 50) {
          — End diff –

          it would be nice, a good way of actually validating everything is behaving as expected

          Show
          githubbot ASF GitHub Bot added a comment - Github user afine commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/157#discussion_r99242422 — Diff: src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java — @@ -839,12 +839,25 @@ public void converseWithFollower(InputArchive ia, OutputArchive oa, Assert.assertEquals(1, f.self.getAcceptedEpoch()); Assert.assertEquals(1, f.self.getCurrentEpoch()); + //Wait for the transactions to be written out. The thread that writes them out + // does not send anything back when it is done. + long start = System.currentTimeMillis(); + while (createSessionZxid != f.fzk.getLastProcessedZxid() && (System.currentTimeMillis() - start) < 50) { — End diff – it would be nice, a good way of actually validating everything is behaving as expected
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user revans2 commented on a diff in the pull request:

          https://github.com/apache/zookeeper/pull/157#discussion_r99347551

          — Diff: src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java —
          @@ -839,12 +839,25 @@ public void converseWithFollower(InputArchive ia, OutputArchive oa,
          Assert.assertEquals(1, f.self.getAcceptedEpoch());
          Assert.assertEquals(1, f.self.getCurrentEpoch());

          + //Wait for the transactions to be written out. The thread that writes them out
          + // does not send anything back when it is done.
          + long start = System.currentTimeMillis();
          + while (createSessionZxid != f.fzk.getLastProcessedZxid() && (System.currentTimeMillis() - start) < 50) {
          — End diff –

          OK Will see what I can do

          Show
          githubbot ASF GitHub Bot added a comment - Github user revans2 commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/157#discussion_r99347551 — Diff: src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java — @@ -839,12 +839,25 @@ public void converseWithFollower(InputArchive ia, OutputArchive oa, Assert.assertEquals(1, f.self.getAcceptedEpoch()); Assert.assertEquals(1, f.self.getCurrentEpoch()); + //Wait for the transactions to be written out. The thread that writes them out + // does not send anything back when it is done. + long start = System.currentTimeMillis(); + while (createSessionZxid != f.fzk.getLastProcessedZxid() && (System.currentTimeMillis() - start) < 50) { — End diff – OK Will see what I can do
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hanm commented on the issue:

          https://github.com/apache/zookeeper/pull/157

          Had another look at the patch, specifically the changes on Learner. Looks good to me. +1.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/157 Had another look at the patch, specifically the changes on Learner. Looks good to me. +1.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user revans2 commented on the issue:

          https://github.com/apache/zookeeper/pull/157

          @afine I updated the test to spy on the `LearnerZooKeeperServer` instance and check if and when `takeSnapshot` was called. I think this fulfills your desires, but with tests there can always be more. So, if you do want more tests or other areas covered please let me know.

          Show
          githubbot ASF GitHub Bot added a comment - Github user revans2 commented on the issue: https://github.com/apache/zookeeper/pull/157 @afine I updated the test to spy on the `LearnerZooKeeperServer` instance and check if and when `takeSnapshot` was called. I think this fulfills your desires, but with tests there can always be more. So, if you do want more tests or other areas covered please let me know.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. GitHub Pull Request Build

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/281//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/281//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/281//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/281//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/281//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/281//console This message is automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user afine commented on the issue:

          https://github.com/apache/zookeeper/pull/157

          Thanks @revans2

          +1 lgtm

          Show
          githubbot ASF GitHub Bot added a comment - Github user afine commented on the issue: https://github.com/apache/zookeeper/pull/157 Thanks @revans2 +1 lgtm
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user revans2 commented on the issue:

          https://github.com/apache/zookeeper/pull/157

          Is there any more I need to do to get this merged in?

          Show
          githubbot ASF GitHub Bot added a comment - Github user revans2 commented on the issue: https://github.com/apache/zookeeper/pull/157 Is there any more I need to do to get this merged in?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hanm commented on the issue:

          https://github.com/apache/zookeeper/pull/157

          @revans2 No more work is required, the patch is ready, but I am not sure if this should be included in the upcoming 3.4.10 release. If not we will wait until 3.4.10 is out to merge this into branch-3.4. @rakeshadr Do you think this should be included in 3.4.10?

          The PR to master https://github.com/apache/zookeeper/pull/159/ could be merged in, I'll have another look and merge it today.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/157 @revans2 No more work is required, the patch is ready, but I am not sure if this should be included in the upcoming 3.4.10 release. If not we will wait until 3.4.10 is out to merge this into branch-3.4. @rakeshadr Do you think this should be included in 3.4.10? The PR to master https://github.com/apache/zookeeper/pull/159/ could be merged in, I'll have another look and merge it today.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hanm commented on a diff in the pull request:

          https://github.com/apache/zookeeper/pull/159#discussion_r100389617

          — Diff: src/java/main/org/apache/zookeeper/server/quorum/Learner.java —
          @@ -498,14 +504,19 @@ else if (qp.getType() == Leader.SNAP)

          { throw new Exception("changes proposed in reconfig"); }

          }

          • if (!snapshotTaken) { // true for the pre v1.0 case
          • zk.takeSnapshot();
            + if (isPreZAB1_0) { + zk.takeSnapshot(); self.setCurrentEpoch(newEpoch); }

            self.setZooKeeperServer(zk);
            self.adminServer.setZooKeeperServer(zk);
            break outerLoop;

          • case Leader.NEWLEADER: // it will be NEWLEADER in v1.0
            + case Leader.NEWLEADER: // Getting NEWLEADER here instead of in discovery
            + // means this is Zab 1.0
            + // Create updatingEpoch file and remove it after current
              • End diff –

          This comment does not apply here - my guess is it is a copy paste error from your branch-3.4 patch... can we get this comment removed?

          Show
          githubbot ASF GitHub Bot added a comment - Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/159#discussion_r100389617 — Diff: src/java/main/org/apache/zookeeper/server/quorum/Learner.java — @@ -498,14 +504,19 @@ else if (qp.getType() == Leader.SNAP) { throw new Exception("changes proposed in reconfig"); } } if (!snapshotTaken) { // true for the pre v1.0 case zk.takeSnapshot(); + if (isPreZAB1_0) { + zk.takeSnapshot(); self.setCurrentEpoch(newEpoch); } self.setZooKeeperServer(zk); self.adminServer.setZooKeeperServer(zk); break outerLoop; case Leader.NEWLEADER: // it will be NEWLEADER in v1.0 + case Leader.NEWLEADER: // Getting NEWLEADER here instead of in discovery + // means this is Zab 1.0 + // Create updatingEpoch file and remove it after current End diff – This comment does not apply here - my guess is it is a copy paste error from your branch-3.4 patch... can we get this comment removed?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user revans2 commented on a diff in the pull request:

          https://github.com/apache/zookeeper/pull/159#discussion_r100552192

          — Diff: src/java/main/org/apache/zookeeper/server/quorum/Learner.java —
          @@ -498,14 +504,19 @@ else if (qp.getType() == Leader.SNAP)

          { throw new Exception("changes proposed in reconfig"); }

          }

          • if (!snapshotTaken) { // true for the pre v1.0 case
          • zk.takeSnapshot();
            + if (isPreZAB1_0) { + zk.takeSnapshot(); self.setCurrentEpoch(newEpoch); }

            self.setZooKeeperServer(zk);
            self.adminServer.setZooKeeperServer(zk);
            break outerLoop;

          • case Leader.NEWLEADER: // it will be NEWLEADER in v1.0
            + case Leader.NEWLEADER: // Getting NEWLEADER here instead of in discovery
            + // means this is Zab 1.0
            + // Create updatingEpoch file and remove it after current
              • End diff –

          You are right will fix that.

          Show
          githubbot ASF GitHub Bot added a comment - Github user revans2 commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/159#discussion_r100552192 — Diff: src/java/main/org/apache/zookeeper/server/quorum/Learner.java — @@ -498,14 +504,19 @@ else if (qp.getType() == Leader.SNAP) { throw new Exception("changes proposed in reconfig"); } } if (!snapshotTaken) { // true for the pre v1.0 case zk.takeSnapshot(); + if (isPreZAB1_0) { + zk.takeSnapshot(); self.setCurrentEpoch(newEpoch); } self.setZooKeeperServer(zk); self.adminServer.setZooKeeperServer(zk); break outerLoop; case Leader.NEWLEADER: // it will be NEWLEADER in v1.0 + case Leader.NEWLEADER: // Getting NEWLEADER here instead of in discovery + // means this is Zab 1.0 + // Create updatingEpoch file and remove it after current End diff – You are right will fix that.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user revans2 commented on the issue:

          https://github.com/apache/zookeeper/pull/159

          @hanm thanks for the reviews I removed the incorrect comment.

          Show
          githubbot ASF GitHub Bot added a comment - Github user revans2 commented on the issue: https://github.com/apache/zookeeper/pull/159 @hanm thanks for the reviews I removed the incorrect comment.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. GitHub Pull Request Build

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/295//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/295//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/295//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/295//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/295//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/295//console This message is automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/zookeeper/pull/159

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/zookeeper/pull/159
          Hide
          hanm Michael Han added a comment -

          Failed tests on pre commit are not related to the pull requested. Merged to master https://github.com/apache/zookeeper/commit/bbfd0169417c316bbdd22ff7a38176845aa0efc6. Thanks for your patch Robert Joseph Evans!
          JIRA not resolved due to pending request https://github.com/apache/zookeeper/pull/157 to branch-3.4.

          Show
          hanm Michael Han added a comment - Failed tests on pre commit are not related to the pull requested. Merged to master https://github.com/apache/zookeeper/commit/bbfd0169417c316bbdd22ff7a38176845aa0efc6 . Thanks for your patch Robert Joseph Evans ! JIRA not resolved due to pending request https://github.com/apache/zookeeper/pull/157 to branch-3.4.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build ZooKeeper-trunk #3273 (See https://builds.apache.org/job/ZooKeeper-trunk/3273/)
          ZOOKEEPER-2678: Discovery and Sync can take a very long time on large (hanm: rev bbfd0169417c316bbdd22ff7a38176845aa0efc6)

          • (edit) src/java/main/org/apache/zookeeper/server/quorum/Learner.java
          • (edit) src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java
          • (edit) src/java/main/org/apache/zookeeper/server/ZooKeeperServerMain.java
          • (edit) src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build ZooKeeper-trunk #3273 (See https://builds.apache.org/job/ZooKeeper-trunk/3273/ ) ZOOKEEPER-2678 : Discovery and Sync can take a very long time on large (hanm: rev bbfd0169417c316bbdd22ff7a38176845aa0efc6) (edit) src/java/main/org/apache/zookeeper/server/quorum/Learner.java (edit) src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java (edit) src/java/main/org/apache/zookeeper/server/ZooKeeperServerMain.java (edit) src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rakeshadr commented on the issue:

          https://github.com/apache/zookeeper/pull/157

          @hanm , mostly it makes sense to me as you all have done detailed code reviews and there is no open comments now. I hope the code path is thoroughly discussed/tested? I'd like to freeze the code changes asap.

          Show
          githubbot ASF GitHub Bot added a comment - Github user rakeshadr commented on the issue: https://github.com/apache/zookeeper/pull/157 @hanm , mostly it makes sense to me as you all have done detailed code reviews and there is no open comments now. I hope the code path is thoroughly discussed/tested? I'd like to freeze the code changes asap.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user revans2 commented on the issue:

          https://github.com/apache/zookeeper/pull/157

          @rakeshadr If it makes you feel any better we have been running with an older version of this patch in production for a while. We have used it as part of a rolling upgrade at least 10 times in production where if it were not there we would have had some very painful outages.

          I have also manually tested it at least 50 times shooting the leader under load (10,000 operations/second) on a 3.4 GB DB, watching it recover, and then validating the integrity of the DB to be sure we didn't get any corruption.

          Show
          githubbot ASF GitHub Bot added a comment - Github user revans2 commented on the issue: https://github.com/apache/zookeeper/pull/157 @rakeshadr If it makes you feel any better we have been running with an older version of this patch in production for a while. We have used it as part of a rolling upgrade at least 10 times in production where if it were not there we would have had some very painful outages. I have also manually tested it at least 50 times shooting the leader under load (10,000 operations/second) on a 3.4 GB DB, watching it recover, and then validating the integrity of the DB to be sure we didn't get any corruption.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rakeshadr commented on the issue:

          https://github.com/apache/zookeeper/pull/157

          Thanks @revans2 for the useful results. I spend sometime going through the changes and looks good to me. @hanm, please go ahead with merging this to branch-3.4.

          Show
          githubbot ASF GitHub Bot added a comment - Github user rakeshadr commented on the issue: https://github.com/apache/zookeeper/pull/157 Thanks @revans2 for the useful results. I spend sometime going through the changes and looks good to me. @hanm, please go ahead with merging this to branch-3.4.
          Hide
          hanm Michael Han added a comment -

          Issue resolved by pull request 157
          https://github.com/apache/zookeeper/pull/157

          Show
          hanm Michael Han added a comment - Issue resolved by pull request 157 https://github.com/apache/zookeeper/pull/157
          Show
          hanm Michael Han added a comment - Merged to 3.4: https://github.com/apache/zookeeper/commit/2de93fe450b909ff76ac8bd8fa44296a515845a5 . cc Rakesh R .
          Hide
          revans2 Robert Joseph Evans added a comment -

          Thanks everyone. I appreciate the great feedback and the quick turnaround time on this.

          Show
          revans2 Robert Joseph Evans added a comment - Thanks everyone. I appreciate the great feedback and the quick turnaround time on this.
          Hide
          rakeshr Rakesh R added a comment -

          Hi Michael Han, I think this is not merged into branch-3.5. I failed to see any comments about branch-3.5 merge discussion, was that purposefully skipped?

          Show
          rakeshr Rakesh R added a comment - Hi Michael Han , I think this is not merged into branch-3.5 . I failed to see any comments about branch-3.5 merge discussion, was that purposefully skipped?
          Hide
          hanm Michael Han added a comment - - edited

          Rakesh R - yes it was deliberately skipped as I was not sure whether to put this in any releasing branch or not (that was before the discussions on whether or not we put this in 3.4.10.). Now with this in 3.4.10 I don't see why it should not be included 3.5.3. So I will merge this in 3.5. Thanks for the double check.

          Now merged in branch-3.5
          https://github.com/apache/zookeeper/commit/c164ee454cd70b3e65212ea1367ec86bb16ec022

          Show
          hanm Michael Han added a comment - - edited Rakesh R - yes it was deliberately skipped as I was not sure whether to put this in any releasing branch or not (that was before the discussions on whether or not we put this in 3.4.10.). Now with this in 3.4.10 I don't see why it should not be included 3.5.3. So I will merge this in 3.5. Thanks for the double check. Now merged in branch-3.5 https://github.com/apache/zookeeper/commit/c164ee454cd70b3e65212ea1367ec86bb16ec022
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hanm commented on the issue:

          https://github.com/apache/zookeeper/pull/157

          @revans2 Please close this pull request; it's merged.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/157 @revans2 Please close this pull request; it's merged.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user revans2 closed the pull request at:

          https://github.com/apache/zookeeper/pull/157

          Show
          githubbot ASF GitHub Bot added a comment - Github user revans2 closed the pull request at: https://github.com/apache/zookeeper/pull/157

            People

            • Assignee:
              revans2 Robert Joseph Evans
              Reporter:
              revans2 Robert Joseph Evans
            • Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development