ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-121

SyncRequestProcessor is not closing log stream during shutdown

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0
    • Component/s: server
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The SyncRequestProcessor is not closing log stream during shutdown.

      See FIXMEs with this ID in ClientBase.java – I've commented out the assertion for the time being (checking for logs being deleted), as part of this fix re-enable these asserts and also verify tests on a Windows system.

      1. ZOOKEEPER-121.patch
        7 kB
        Mahadev konar
      2. ZOOKEEPER-121.patch
        7 kB
        Mahadev konar
      3. ZOOKEEPER-121.patch
        1 kB
        Mahadev konar
      4. ZOOKEEPER-121.patch
        0.8 kB
        Benjamin Reed
      5. ZOOKEEPER-121_4.patch
        3 kB
        Mahadev konar
      6. ZOOKEEPER-121_3.patch
        0.6 kB
        Mahadev konar
      7. ZOOKEEPER-121_3.patch
        4 kB
        Mahadev konar
      8. ZOOKEEPER-121_3.patch
        4 kB
        Mahadev konar

        Issue Links

          Activity

          Hide
          Mahadev konar added a comment -

          this patch closes the streams on a shutdown of syncrequestprocessor.

          Show
          Mahadev konar added a comment - this patch closes the streams on a shutdown of syncrequestprocessor.
          Hide
          Mahadev konar added a comment -

          minor change, freeing snapshot resources as well.

          Show
          Mahadev konar added a comment - minor change, freeing snapshot resources as well.
          Hide
          Benjamin Reed added a comment -

          +1 looks good

          Show
          Benjamin Reed added a comment - +1 looks good
          Hide
          Mahadev konar added a comment -

          running hudson...

          Show
          Mahadev konar added a comment - running hudson...
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12438266/ZOOKEEPER-121.patch
          against trunk revision 922433.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/18/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/18/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/18/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12438266/ZOOKEEPER-121.patch against trunk revision 922433. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/18/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/18/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/18/console This message is automatically generated.
          Hide
          Mahadev konar added a comment -

          I just committed this.

          Show
          Mahadev konar added a comment - I just committed this.
          Hide
          Patrick Hunt added a comment -

          looks to me like this change is causing the build to fail:

          [junit] 2010-03-13 02:09:18,679 - FATAL [SyncThread:1:SyncRequestProcessor@148] - Severe unrecoverable error, exiting
          [junit] java.nio.channels.ClosedChannelException
          [junit] at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91)
          [junit] at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:256)
          [junit] at org.apache.zookeeper.server.persistence.Util.padLogFile(Util.java:214)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.padFile(FileTxnLog.java:228)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:206)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:269)
          [junit] at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:435)
          [junit] at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:107)

          Show
          Patrick Hunt added a comment - looks to me like this change is causing the build to fail: [junit] 2010-03-13 02:09:18,679 - FATAL [SyncThread:1:SyncRequestProcessor@148] - Severe unrecoverable error, exiting [junit] java.nio.channels.ClosedChannelException [junit] at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91) [junit] at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:256) [junit] at org.apache.zookeeper.server.persistence.Util.padLogFile(Util.java:214) [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.padFile(FileTxnLog.java:228) [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:206) [junit] at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:269) [junit] at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:435) [junit] at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:107)
          Hide
          Mahadev konar added a comment -

          this patch should fix the issue.

          Show
          Mahadev konar added a comment - this patch should fix the issue.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12438711/ZOOKEEPER-121.patch
          against trunk revision 922459.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/21/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/21/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/21/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12438711/ZOOKEEPER-121.patch against trunk revision 922459. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/21/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/21/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/21/console This message is automatically generated.
          Hide
          Patrick Hunt added a comment -

          I don't get this patch, more explanation is necessary.

          1) first, why are we calling append on a closed log? in what situations?

          2) more importantly, this patch seems to ignore the problem, in particular it results in append being called, append ignores (granted warn log) the operation and return immediately. How is this acceptable? ie we are asked to append and it seems we essentially drop the request and continue on (in syncrequestprocessor run method loop)

          Show
          Patrick Hunt added a comment - I don't get this patch, more explanation is necessary. 1) first, why are we calling append on a closed log? in what situations? 2) more importantly, this patch seems to ignore the problem, in particular it results in append being called, append ignores (granted warn log) the operation and return immediately. How is this acceptable? ie we are asked to append and it seems we essentially drop the request and continue on (in syncrequestprocessor run method loop)
          Hide
          Benjamin Reed added a comment -

          i think i agree with pat. we need to figure why it is happening.

          looking more closely i believe the problem is in SyncRequestProcessor.shutdown.

          currently it is

              public void shutdown() {
                  try{
                      zks.getZKDatabase().close();
                  } catch (IOException ie) {
                      LOG.warn("Error closing logs ", ie);
                  }
                  queuedRequests.add(requestOfDeath);
                  nextProcessor.shutdown();
              }
          

          i think we need to move up the requestOfDeath and wait for shutdown before closing:

              public void shutdown() {
                  queuedRequests.add(requestOfDeath);
                  this.join();
                  try{
                      zks.getZKDatabase().close();
                  } catch (IOException ie) {
                      LOG.warn("Error closing logs ", ie);
                  }
                  nextProcessor.shutdown();
              }
          
          Show
          Benjamin Reed added a comment - i think i agree with pat. we need to figure why it is happening. looking more closely i believe the problem is in SyncRequestProcessor.shutdown. currently it is public void shutdown() { try { zks.getZKDatabase().close(); } catch (IOException ie) { LOG.warn( "Error closing logs " , ie); } queuedRequests.add(requestOfDeath); nextProcessor.shutdown(); } i think we need to move up the requestOfDeath and wait for shutdown before closing: public void shutdown() { queuedRequests.add(requestOfDeath); this .join(); try { zks.getZKDatabase().close(); } catch (IOException ie) { LOG.warn( "Error closing logs " , ie); } nextProcessor.shutdown(); }
          Hide
          Benjamin Reed added a comment -

          the patch that corresponds to my comment.

          Show
          Benjamin Reed added a comment - the patch that corresponds to my comment.
          Hide
          Mahadev konar added a comment -

          agreed. good point folks.... that was a hurried up patch from my side. sorry abt that.... I should have been more careful!

          Show
          Mahadev konar added a comment - agreed. good point folks.... that was a hurried up patch from my side. sorry abt that.... I should have been more careful!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12438729/ZOOKEEPER-121.patch
          against trunk revision 922459.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/22/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/22/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/22/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12438729/ZOOKEEPER-121.patch against trunk revision 922459. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/22/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/22/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/22/console This message is automatically generated.
          Hide
          Mahadev konar added a comment -

          +1 the patch looks good....

          Show
          Mahadev konar added a comment - +1 the patch looks good....
          Hide
          Mahadev konar added a comment -

          I just committed this. thanks ben!

          Show
          Mahadev konar added a comment - I just committed this. thanks ben!
          Hide
          Patrick Hunt added a comment -

          still seeing this fatal even after applying latest patch. plz name the patches _2 _3 etc... so they aren't superceeded by jira.

          [junit] 2010-03-14 05:38:06,999 - FATAL [SyncThread:3:SyncRequestProcessor@148] - Severe unrecoverable error, exiting
          [junit] java.nio.channels.ClosedChannelException
          [junit] at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91)
          [junit] at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:256)
          [junit] at org.apache.zookeeper.server.persistence.Util.padLogFile(Util.java:214)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.padFile(FileTxnLog.java:228)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:206)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:269)
          [junit] at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:435)
          [junit] at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:107)

          Show
          Patrick Hunt added a comment - still seeing this fatal even after applying latest patch. plz name the patches _2 _3 etc... so they aren't superceeded by jira. [junit] 2010-03-14 05:38:06,999 - FATAL [SyncThread:3:SyncRequestProcessor@148] - Severe unrecoverable error, exiting [junit] java.nio.channels.ClosedChannelException [junit] at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91) [junit] at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:256) [junit] at org.apache.zookeeper.server.persistence.Util.padLogFile(Util.java:214) [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.padFile(FileTxnLog.java:228) [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:206) [junit] at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:269) [junit] at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:435) [junit] at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:107)
          Hide
          Benjamin Reed added a comment -

          you should apply just one of the patches not both. it sounds like you applied both of them pat.

          Show
          Benjamin Reed added a comment - you should apply just one of the patches not both. it sounds like you applied both of them pat.
          Hide
          Mahadev konar added a comment -

          ben,
          only your patch was applied. Also, the output is from a trunk run after the patch was committed!

          Show
          Mahadev konar added a comment - ben, only your patch was applied. Also, the output is from a trunk run after the patch was committed!
          Hide
          Patrick Hunt added a comment -

          My comment was that mahadev should have named his patch ..._2.patch (vs the original patch that was applied before this issue was reopened, ie the fix for the original problem not the current blocker)

          Show
          Patrick Hunt added a comment - My comment was that mahadev should have named his patch ..._2.patch (vs the original patch that was applied before this issue was reopened, ie the fix for the original problem not the current blocker)
          Hide
          Hudson added a comment -

          Integrated in ZooKeeper-trunk #738 (See http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/738/)

          Show
          Hudson added a comment - Integrated in ZooKeeper-trunk #738 (See http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/738/ )
          Hide
          Patrick Hunt added a comment -

          Still failing in some cases, most recent:

          [junit] 2010-03-16 00:39:13,346 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:11657:Follower@116] - Got zxid 0x100000059 expected 0x1
          [junit] 2010-03-16 00:39:13,353 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:11657:FileTxnSnapLog@208] - Snapshotting: 100000153
          [junit] 2010-03-16 00:39:13,354 - FATAL [SyncThread:4:SyncRequestProcessor@148] - Severe unrecoverable error, exiting
          [junit] java.nio.channels.ClosedChannelException
          [junit] at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91)
          [junit] at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:256)
          [junit] at org.apache.zookeeper.server.persistence.Util.padLogFile(Util.java:214)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.padFile(FileTxnLog.java:228)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:206)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:269)
          [junit] at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:435)
          [junit] at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:107)
          [junit] 2010-03-16 00:39:13,355 - FATAL [SyncThread:3:SyncRequestProcessor@148] - Severe unrecoverable error, exiting
          [junit] java.nio.channels.ClosedChannelException
          [junit] at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91)
          [junit] at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:256)
          [junit] at org.apache.zookeeper.server.persistence.Util.padLogFile(Util.java:214)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.padFile(FileTxnLog.java:228)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:206)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:269)
          [junit] at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:435)
          [junit] at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:107)
          [junit] 2010-03-16 00:39:13,357 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:11658:FileTxnSnapLog@208] - Snapshotting: 100000153
          [junit] Running org.apache.zookeeper.test.ZooKeeperQuotaTest
          [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec

          Show
          Patrick Hunt added a comment - Still failing in some cases, most recent: [junit] 2010-03-16 00:39:13,346 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:11657:Follower@116] - Got zxid 0x100000059 expected 0x1 [junit] 2010-03-16 00:39:13,353 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:11657:FileTxnSnapLog@208] - Snapshotting: 100000153 [junit] 2010-03-16 00:39:13,354 - FATAL [SyncThread:4:SyncRequestProcessor@148] - Severe unrecoverable error, exiting [junit] java.nio.channels.ClosedChannelException [junit] at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91) [junit] at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:256) [junit] at org.apache.zookeeper.server.persistence.Util.padLogFile(Util.java:214) [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.padFile(FileTxnLog.java:228) [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:206) [junit] at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:269) [junit] at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:435) [junit] at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:107) [junit] 2010-03-16 00:39:13,355 - FATAL [SyncThread:3:SyncRequestProcessor@148] - Severe unrecoverable error, exiting [junit] java.nio.channels.ClosedChannelException [junit] at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91) [junit] at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:256) [junit] at org.apache.zookeeper.server.persistence.Util.padLogFile(Util.java:214) [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.padFile(FileTxnLog.java:228) [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:206) [junit] at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:269) [junit] at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:435) [junit] at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:107) [junit] 2010-03-16 00:39:13,357 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:11658:FileTxnSnapLog@208] - Snapshotting: 100000153 [junit] Running org.apache.zookeeper.test.ZooKeeperQuotaTest [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
          Hide
          Patrick Hunt added a comment -

          not sure if this helps but anouther failure I see:

          [junit] 2010-03-16 01:12:16,548 - WARN [SyncThread:4:SyncRequestProcessor@115] - Too busy to snap, skipping
          [junit] 2010-03-16 01:12:16,548 - WARN [SyncThread:4:SyncRequestProcessor@115] - Too busy to snap, skipping
          [junit] 2010-03-16 01:12:16,548 - WARN [SyncThread:4:SyncRequestProcessor@115] - Too busy to snap, skipping
          [junit] 2010-03-16 01:12:16,549 - INFO [SyncThread:4:SyncRequestProcessor@151] - SyncRequestProcessor exited!

          then 12ms later

          [junit] 2010-03-16 01:12:16,561 - FATAL [SyncThread:4:SyncRequestProcessor@148] - Severe unrecoverable error, exiting
          [junit] java.nio.channels.ClosedChannelException
          [junit] at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91)
          [junit] at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:256)
          [junit] at org.apache.zookeeper.server.persistence.Util.padLogFile(Util.java:214)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.padFile(FileTxnLog.java:228)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:206)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:269)
          [junit] at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:435)
          [junit] at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:107)

          Show
          Patrick Hunt added a comment - not sure if this helps but anouther failure I see: [junit] 2010-03-16 01:12:16,548 - WARN [SyncThread:4:SyncRequestProcessor@115] - Too busy to snap, skipping [junit] 2010-03-16 01:12:16,548 - WARN [SyncThread:4:SyncRequestProcessor@115] - Too busy to snap, skipping [junit] 2010-03-16 01:12:16,548 - WARN [SyncThread:4:SyncRequestProcessor@115] - Too busy to snap, skipping [junit] 2010-03-16 01:12:16,549 - INFO [SyncThread:4:SyncRequestProcessor@151] - SyncRequestProcessor exited! then 12ms later [junit] 2010-03-16 01:12:16,561 - FATAL [SyncThread:4:SyncRequestProcessor@148] - Severe unrecoverable error, exiting [junit] java.nio.channels.ClosedChannelException [junit] at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91) [junit] at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:256) [junit] at org.apache.zookeeper.server.persistence.Util.padLogFile(Util.java:214) [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.padFile(FileTxnLog.java:228) [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:206) [junit] at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:269) [junit] at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:435) [junit] at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:107)
          Hide
          Mahadev konar added a comment -

          I wasnt able to reproduce the bug on my machine. I am uploading the fix, which might be the reason for the crashes. We need to verify on these machines. Since I cannot reproduce, I have no way to verify this.

          Show
          Mahadev konar added a comment - I wasnt able to reproduce the bug on my machine. I am uploading the fix, which might be the reason for the crashes. We need to verify on these machines. Since I cannot reproduce, I have no way to verify this.
          Hide
          Mahadev konar added a comment -

          hmm, doesnt look like this will fix the issue.

          Show
          Mahadev konar added a comment - hmm, doesnt look like this will fix the issue.
          Hide
          Mahadev konar added a comment -

          this patch should fix the issue. We were closing zkdb in syncrequestprocessor but since it gets shared across zookeeper server instances, we should only be closing it in quorumpeer shutdown. I also added a synchronized in rollLog (htough its not necessary but we should have had it).

          Show
          Mahadev konar added a comment - this patch should fix the issue. We were closing zkdb in syncrequestprocessor but since it gets shared across zookeeper server instances, we should only be closing it in quorumpeer shutdown. I also added a synchronized in rollLog (htough its not necessary but we should have had it).
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12438950/ZOOKEEPER-121_3.patch
          against trunk revision 922759.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/24/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/24/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/24/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12438950/ZOOKEEPER-121_3.patch against trunk revision 922759. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/24/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/24/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/24/console This message is automatically generated.
          Hide
          Mahadev konar added a comment - - edited

          updated patch, fixing if factory is null in ClientBase and moving zkdb.shutdown to be called after factory.shutdown...

          Show
          Mahadev konar added a comment - - edited updated patch, fixing if factory is null in ClientBase and moving zkdb.shutdown to be called after factory.shutdown...
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12438964/ZOOKEEPER-121_3.patch
          against trunk revision 922759.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/26/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/26/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/26/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12438964/ZOOKEEPER-121_3.patch against trunk revision 922759. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/26/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/26/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/26/console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in ZooKeeper-trunk #744 (See http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/744/)
          . SyncRequestProcessor is not closing log stream during shutdown (3rd attempt)

          Show
          Hudson added a comment - Integrated in ZooKeeper-trunk #744 (See http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/744/ ) . SyncRequestProcessor is not closing log stream during shutdown (3rd attempt)
          Hide
          Patrick Hunt added a comment -

          Failed in my testing, this time in
          org.apache.zookeeper.test.ZooKeeperQuotaTest.testCorruption

          [junit] 2010-03-16 21:33:52,868 - FATAL [SyncThread:1:SyncRequestProcessor@148] - Severe unrecoverable error, exiting
          [junit] java.nio.channels.ClosedChannelException
          [junit] at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91)
          [junit] at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:256)
          [junit] at org.apache.zookeeper.server.persistence.Util.padLogFile(Util.java:214)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.padFile(FileTxnLog.java:228)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:206)
          [junit] at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:269)
          [junit] at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:435)
          [junit] at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:107)

          Show
          Patrick Hunt added a comment - Failed in my testing, this time in org.apache.zookeeper.test.ZooKeeperQuotaTest.testCorruption [junit] 2010-03-16 21:33:52,868 - FATAL [SyncThread:1:SyncRequestProcessor@148] - Severe unrecoverable error, exiting [junit] java.nio.channels.ClosedChannelException [junit] at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91) [junit] at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:256) [junit] at org.apache.zookeeper.server.persistence.Util.padLogFile(Util.java:214) [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.padFile(FileTxnLog.java:228) [junit] at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:206) [junit] at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:269) [junit] at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:435) [junit] at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:107)
          Hide
          Mahadev konar added a comment -

          this makes sure, zkdb.close is called after quorumpeer shuts down sync processor.

          Show
          Mahadev konar added a comment - this makes sure, zkdb.close is called after quorumpeer shuts down sync processor.
          Hide
          Benjamin Reed added a comment -

          +1 good catch.

          Show
          Benjamin Reed added a comment - +1 good catch.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12438976/ZOOKEEPER-121_4.patch
          against trunk revision 923984.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 12 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/28/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/28/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/28/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12438976/ZOOKEEPER-121_4.patch against trunk revision 923984. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/28/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/28/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/28/console This message is automatically generated.
          Hide
          Benjamin Reed added a comment -

          Committed revision 924027.

          Show
          Benjamin Reed added a comment - Committed revision 924027.
          Hide
          Hudson added a comment -

          Integrated in ZooKeeper-trunk #746 (See http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/746/)
          . SyncRequestProcessor is not closing log stream during shutdown

          Show
          Hudson added a comment - Integrated in ZooKeeper-trunk #746 (See http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/746/ ) . SyncRequestProcessor is not closing log stream during shutdown

            People

            • Assignee:
              Mahadev konar
              Reporter:
              Patrick Hunt
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development