Cassandra
  1. Cassandra
  2. CASSANDRA-5689

NPE shutting down Cassandra trunk (cassandra-1.2.5-989-g70dfb70)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Trivial Trivial
    • Resolution: Cannot Reproduce
    • Fix Version/s: None
    • Component/s: API
    • Labels:
    • Environment:

      Ubuntu Precise with Oracle Java 7u25.

      Description

      I built Cassandra from git trunk at cassandra-1.2.5-989-g70dfb70 using the debian/ package. I have a shell script to shut down Cassandra:

            $nodetool disablegossip
            sleep 5
            $nodetool disablebinary
            $nodetool disablethrift
            $nodetool drain
            /etc/init.d/cassandra stop
      

      Shutting it down I get this exception on all three nodes:

      Exception in thread "main" java.lang.NullPointerException
      	at org.apache.cassandra.transport.Server.close(Server.java:156)
      	at org.apache.cassandra.transport.Server.stop(Server.java:107)
      	at org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:347)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
      	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
      	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
      	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
      	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
      	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
      	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
      	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
      	at com.sun.jmx.remote.security.MBeanServerAccessController.invoke(MBeanServerAccessController.java:468)
      	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
      	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
      	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1427)
      	at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
      	at sun.rmi.transport.Transport$1.run(Transport.java:177)
      	at sun.rmi.transport.Transport$1.run(Transport.java:174)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
      	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
      	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
      	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:724)
      
      1. init3
        2 kB
        Alex Zarutin
      2. init2
        2 kB
        Alex Zarutin
      3. init1
        2 kB
        Alex Zarutin
      4. CASSANDRA-5689.txt
        16 kB
        Alex Zarutin

        Activity

        Hide
        Jonathan Ellis added a comment -

        I don't see how this can happen since stop should only call close once:

            public void stop()
            {
                if (isRunning.compareAndSet(true, false))
                    close();
            }
        

        Ryan, can one of your team reproduce?

        Show
        Jonathan Ellis added a comment - I don't see how this can happen since stop should only call close once: public void stop() { if (isRunning.compareAndSet( true , false )) close(); } Ryan, can one of your team reproduce?
        Hide
        Alex Zarutin added a comment - - edited

        General comments:
        I use CCM based installed C* cluster, so I do not use init script to stop and start it. So, what I did, I creaated subset of ..../debian/init script to be able to run it from the shell against the cluster started by CCM. Platform is a Mac laptop.

        Number of the tests, including 1 node and 3 nodes cluster did not show any NullPointer or any other exceptions.

        Full repro steps and init1, init2, init3 scripts stripped to the functionality of stopping cluster are attached.

        Show
        Alex Zarutin added a comment - - edited General comments: I use CCM based installed C* cluster, so I do not use init script to stop and start it. So, what I did, I creaated subset of ..../debian/init script to be able to run it from the shell against the cluster started by CCM. Platform is a Mac laptop. Number of the tests, including 1 node and 3 nodes cluster did not show any NullPointer or any other exceptions. Full repro steps and init1, init2, init3 scripts stripped to the functionality of stopping cluster are attached.
        Hide
        Cathy Daw added a comment - - edited

        I was testing the debian installer for DataStax Community 1.2.6 and noticed first that when we install that we do a startup/shutdown, but also that on a debian platform (not centos, rhel, mac or ubuntu) that I consistently see this exception during that initial shutdown. Not sure if this is relevant or not but I was testing on squeeze (debian 6.0.1) using this EC2 AMI: ami-75287b30.

        ERROR [StorageServiceShutdownHook] 2013-07-03 00:35:16,628 CassandraDaemon.java (line 192) Exception in thread Thread[StorageServiceShutdownHook,5,main]
        java.lang.NullPointerException
                at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:321)
                at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:370)
                at org.apache.cassandra.service.StorageService.access$000(StorageService.java:88)
                at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:519)
                at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
                at java.lang.Thread.run(Thread.java:662)
        
        Show
        Cathy Daw added a comment - - edited I was testing the debian installer for DataStax Community 1.2.6 and noticed first that when we install that we do a startup/shutdown, but also that on a debian platform (not centos, rhel, mac or ubuntu) that I consistently see this exception during that initial shutdown. Not sure if this is relevant or not but I was testing on squeeze (debian 6.0.1) using this EC2 AMI: ami-75287b30. ERROR [StorageServiceShutdownHook] 2013-07-03 00:35:16,628 CassandraDaemon.java (line 192) Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:321) at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:370) at org.apache.cassandra.service.StorageService.access$000(StorageService.java:88) at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:519) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.lang.Thread.run(Thread.java:662)
        Hide
        Blair Zajac added a comment -

        I'm running 1.2.5 using DataStax's Debian cassandra package and using the same shutdown script, I haven't seen this exception at all through many shutdowns on my VMs, so this suggests it's new in 1.2.6 and trunk.

        Show
        Blair Zajac added a comment - I'm running 1.2.5 using DataStax's Debian cassandra package and using the same shutdown script, I haven't seen this exception at all through many shutdowns on my VMs, so this suggests it's new in 1.2.6 and trunk.
        Hide
        Cathy Daw added a comment - - edited

        FWIW, I hadn't seen it testing the 1.2.5 installer either. In my tests, I have been using java 1.6_0.45.

        Show
        Cathy Daw added a comment - - edited FWIW, I hadn't seen it testing the 1.2.5 installer either. In my tests, I have been using java 1.6_0.45.
        Hide
        Cathy Daw added a comment -

        Brandon Williams
        I was not able to reproduce this using the Apache Cassandra 1.2.6 packages, via these instructions: http://wiki.apache.org/cassandra/DebianPackaging

        First Test: I installed, started up the server, and manually stopped the server:
        sudo service cassandra stop

        Second Test: start/stop a few times
        sudo service cassandra start
        sudo service cassandra stop

        Third Test:
        $ nodetool disablegossip
        $ nodetool disablebinary
        $ nodetool disablethrift
        $ nodetool drain
        $ sudo /etc/init.d/cassandra stop

        Show
        Cathy Daw added a comment - Brandon Williams I was not able to reproduce this using the Apache Cassandra 1.2.6 packages, via these instructions: http://wiki.apache.org/cassandra/DebianPackaging First Test: I installed, started up the server, and manually stopped the server: sudo service cassandra stop Second Test: start/stop a few times sudo service cassandra start sudo service cassandra stop Third Test: $ nodetool disablegossip $ nodetool disablebinary $ nodetool disablethrift $ nodetool drain $ sudo /etc/init.d/cassandra stop
        Hide
        Cathy Daw added a comment -

        I'll raise a bug in the DSE bug system to debug more.

        Show
        Cathy Daw added a comment - I'll raise a bug in the DSE bug system to debug more.

          People

          • Assignee:
            Cathy Daw
            Reporter:
            Blair Zajac
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development