ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-547

Sanity check in QuorumCnxn Manager and quorum communication port.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.0, 3.2.1
    • Fix Version/s: 3.2.2, 3.3.0
    • Component/s: leaderElection, server
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We need to put some sanity checks in QuorumCnxnManager and the other quorum port for rogue clients. Sometimes a clients might get misconfigured and they might send random characters on such ports. We need to make sure that such rogue clients do not bring down the clients and need to put in some sanity checks with respect to packet lengths and deserialization.

      1. ZOOKEEPER-547.patch
        1 kB
        Mahadev konar
      2. ZOOKEEPER-547.patch
        4 kB
        Mahadev konar
      3. ZOOKEEPER-547.patch
        4 kB
        Mahadev konar
      4. ZOOKEEPER-547.patch
        5 kB
        Mahadev konar

        Activity

        Hide
        Mahadev konar added a comment -

        stack trace with such rogue packets:

        2009-10-08 19:47:35,063 - WARN  [NIOServerCxn.Factory:2281:NIOServerCnxn@494] - Exception causing close of session 0x0 due to java.io.IOException: Len error -720899
        2009-10-08 19:47:35,064 - INFO  [NIOServerCxn.Factory:2281:NIOServerCnxn@833] - closing session:0x0 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/***********:2281 remote=/*******:33530]
        2009-10-08 19:47:41,961 - ERROR [FollowerHandler-/*************:38326:FollowerHandler@415] - Unexpected exception causing shutdown while sock still open
        java.io.IOException: Unreasonable length = -195059
                at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:100)
                at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:68)
                at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
                at org.apache.zookeeper.server.quorum.FollowerHandler.run(FollowerHandler.java:215)
        2009-10-08 19:47:41,962 - WARN  [FollowerHandler-/***********:38326:FollowerHandler@421] - ******* GOODBYE /***********:38326 ********
        2009-10-08 19:49:19,409 - WARN  [Thread-6:QuorumCnxManager$SendWorker@539] - Interrupted while waiting for message on queue
        java.lang.InterruptedException
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1934)
                at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
                at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:533)
        2009-10-08 19:49:19,410 - WARN  [Thread-6:QuorumCnxManager$SendWorker@554] - Send worker leaving thread
        2009-10-08 19:49:19,418 - WARN  [Thread-7:QuorumCnxManager$RecvWorker@621] - Connection broken: 
        java.nio.channels.AsynchronousCloseException
                at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
                at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
                at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:609)
        Exception in thread "Thread-9" java.lang.OutOfMemoryError: Java heap space
                at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:605)
        
        Show
        Mahadev konar added a comment - stack trace with such rogue packets: 2009-10-08 19:47:35,063 - WARN [NIOServerCxn.Factory:2281:NIOServerCnxn@494] - Exception causing close of session 0x0 due to java.io.IOException: Len error -720899 2009-10-08 19:47:35,064 - INFO [NIOServerCxn.Factory:2281:NIOServerCnxn@833] - closing session:0x0 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/***********:2281 remote=/*******:33530] 2009-10-08 19:47:41,961 - ERROR [FollowerHandler-/*************:38326:FollowerHandler@415] - Unexpected exception causing shutdown while sock still open java.io.IOException: Unreasonable length = -195059 at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:100) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:68) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108) at org.apache.zookeeper.server.quorum.FollowerHandler.run(FollowerHandler.java:215) 2009-10-08 19:47:41,962 - WARN [FollowerHandler-/***********:38326:FollowerHandler@421] - ******* GOODBYE /***********:38326 ******** 2009-10-08 19:49:19,409 - WARN [ Thread -6:QuorumCnxManager$SendWorker@539] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1934) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:533) 2009-10-08 19:49:19,410 - WARN [ Thread -6:QuorumCnxManager$SendWorker@554] - Send worker leaving thread 2009-10-08 19:49:19,418 - WARN [ Thread -7:QuorumCnxManager$RecvWorker@621] - Connection broken: java.nio.channels.AsynchronousCloseException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:609) Exception in thread " Thread -9" java.lang.OutOfMemoryError: Java heap space at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:605)
        Hide
        Mahadev konar added a comment -

        a simple fix. We are already guarded on our client and followerhandler ports. This one gurads against the quurom ports. I did write a test case, but havent been able to get it working since we unfortunately do not use FLE in most of our testing. I will try and upload a test case within a day!

        Show
        Mahadev konar added a comment - a simple fix. We are already guarded on our client and followerhandler ports. This one gurads against the quurom ports. I did write a test case, but havent been able to get it working since we unfortunately do not use FLE in most of our testing. I will try and upload a test case within a day!
        Hide
        Mahadev konar added a comment -

        added a testcase in QuorumpeerMainTest.

        Show
        Mahadev konar added a comment - added a testcase in QuorumpeerMainTest.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12423075/ZOOKEEPER-547.patch
        against trunk revision 828216.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/37/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/37/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/37/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423075/ZOOKEEPER-547.patch against trunk revision 828216. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/37/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/37/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/37/console This message is automatically generated.
        Hide
        Mahadev konar added a comment -

        a minor change in the test so that buff is set to 0 when writing out invalid packets.

        Show
        Mahadev konar added a comment - a minor change in the test so that buff is set to 0 when writing out invalid packets.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12423080/ZOOKEEPER-547.patch
        against trunk revision 828216.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/38/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/38/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/38/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423080/ZOOKEEPER-547.patch against trunk revision 828216. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/38/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/38/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/38/console This message is automatically generated.
        Hide
        Patrick Hunt added a comment -

        Why did the unit tests fail?

        Show
        Patrick Hunt added a comment - Why did the unit tests fail?
        Hide
        Mahadev konar added a comment -

        saw this in the logs -

        junit.framework.AssertionFailedError: fastleaderelection used
        	at org.apache.zookeeper.server.quorum.QuorumPeerMainTest.testQuorumDefaults(QuorumPeerMainTest.java:354)
        

        Doesnt seem to be related to the patch but will have to take a closer look.

        Show
        Mahadev konar added a comment - saw this in the logs - junit.framework.AssertionFailedError: fastleaderelection used at org.apache.zookeeper.server.quorum.QuorumPeerMainTest.testQuorumDefaults(QuorumPeerMainTest.java:354) Doesnt seem to be related to the patch but will have to take a closer look.
        Hide
        Flavio Junqueira added a comment -

        The log also says this:

        Exception in thread "WorkerSender Thread" java.nio.channels.UnresolvedAddressException
        	at sun.nio.ch.Net.checkAddress(Net.java:30)
        	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:487)
        	at java.nio.channels.SocketChannel.open(SocketChannel.java:146)
        	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:323)
        	at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:302)
        	at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:323)
        	at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:296)
        	at java.lang.Thread.run(Thread.java:619)
        
        Show
        Flavio Junqueira added a comment - The log also says this: Exception in thread "WorkerSender Thread " java.nio.channels.UnresolvedAddressException at sun.nio.ch.Net.checkAddress(Net.java:30) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:487) at java.nio.channels.SocketChannel.open(SocketChannel.java:146) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:323) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:302) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:323) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:296) at java.lang. Thread .run( Thread .java:619)
        Hide
        Patrick Hunt added a comment -

        Let's give it another go and see, perhaps some hiccup on h8

        Show
        Patrick Hunt added a comment - Let's give it another go and see, perhaps some hiccup on h8
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12423080/ZOOKEEPER-547.patch
        against trunk revision 833559.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/53/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/53/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/53/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423080/ZOOKEEPER-547.patch against trunk revision 833559. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/53/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/53/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/53/console This message is automatically generated.
        Hide
        Flavio Junqueira added a comment -

        Same problem as before:

        Exception in thread "WorkerSender Thread" java.nio.channels.UnresolvedAddressException
        	at sun.nio.ch.Net.checkAddress(Net.java:30)
        	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:487)
        	at java.nio.channels.SocketChannel.open(SocketChannel.java:146)
        	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:330)
        	at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:303)
        	at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:323)
        	at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:296)
        	at java.lang.Thread.run(Thread.java:619)
        
        
        Show
        Flavio Junqueira added a comment - Same problem as before: Exception in thread "WorkerSender Thread " java.nio.channels.UnresolvedAddressException at sun.nio.ch.Net.checkAddress(Net.java:30) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:487) at java.nio.channels.SocketChannel.open(SocketChannel.java:146) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:330) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:303) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:323) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:296) at java.lang. Thread .run( Thread .java:619)
        Hide
        Mahadev konar added a comment -

        this patch fixes the test case to pass.

        Show
        Mahadev konar added a comment - this patch fixes the test case to pass.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12425168/ZOOKEEPER-547.patch
        against trunk revision 835618.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/66/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/66/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/66/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12425168/ZOOKEEPER-547.patch against trunk revision 835618. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/66/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/66/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/66/console This message is automatically generated.
        Hide
        Benjamin Reed added a comment -

        +1 looks good.

        Show
        Benjamin Reed added a comment - +1 looks good.
        Hide
        Flavio Junqueira added a comment -

        +1

        Show
        Flavio Junqueira added a comment - +1
        Hide
        Benjamin Reed added a comment -

        Committed revision 881623.

        Show
        Benjamin Reed added a comment - Committed revision 881623.
        Hide
        Benjamin Reed added a comment -

        Committed revision 881641. (for branch 3.2.2) i had to pull in src/java/test/org/apache/zookeeper/PortAssignment.java
        from trunk.

        Show
        Benjamin Reed added a comment - Committed revision 881641. (for branch 3.2.2) i had to pull in src/java/test/org/apache/zookeeper/PortAssignment.java from trunk.
        Hide
        Hudson added a comment -

        Integrated in ZooKeeper-trunk #543 (See http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/543/)
        . Sanity check in QuorumCnxn Manager and quorum communication port.

        Show
        Hudson added a comment - Integrated in ZooKeeper-trunk #543 (See http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/543/ ) . Sanity check in QuorumCnxn Manager and quorum communication port.

          People

          • Assignee:
            Mahadev konar
            Reporter:
            Mahadev konar
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development