Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-13058

dtest failure in hintedhandoff_test.TestHintedHandoff.hintedhandoff_decom_test

    XMLWordPrintableJSON

Details

    Description

      example failure:

      http://cassci.datastax.com/job/cassandra-3.X_novnode_dtest/16/testReport/hintedhandoff_test/TestHintedHandoff/hintedhandoff_decom_test/

      Error Message
      
      Subprocess ['nodetool', '-h', 'localhost', '-p', '7100', ['decommission']] exited with non-zero status; exit status: 2; 
      stderr: error: Error while decommissioning node: Failed to transfer all hints to 59f20b4f-0215-4e18-be1b-7e00f2901629
      
      Stacktrace
      
        File "/usr/lib/python2.7/unittest/case.py", line 329, in run
          testMethod()
        File "/home/automaton/cassandra-dtest/hintedhandoff_test.py", line 167, in hintedhandoff_decom_test
          node1.decommission()
        File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1314, in decommission
          self.nodetool("decommission")
        File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 783, in nodetool
          return handle_external_tool_process(p, ['nodetool', '-h', 'localhost', '-p', str(self.jmx_port), cmd.split()])
        File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 1993, in handle_external_tool_process
          raise ToolError(cmd_args, rc, out, err)
      
      java.lang.RuntimeException: Error while decommissioning node: Failed to transfer all hints to 59f20b4f-0215-4e18-be1b-7e00f2901629
      	at org.apache.cassandra.service.StorageService.decommission(StorageService.java:3924)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
      	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
      	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
      	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
      	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
      	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
      	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
      	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1466)
      	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
      	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1307)
      	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1399)
      	at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:828)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
      	at sun.rmi.transport.Transport$1.run(Transport.java:200)
      	at sun.rmi.transport.Transport$1.run(Transport.java:197)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
      	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
      	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
      	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$241(TCPTransport.java:683)
      	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$284/1694175644.run(Unknown Source)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      Attachments

        1. node4.log
          38 kB
          Sean McCarthy
        2. node4_gc.log
          19 kB
          Sean McCarthy
        3. node4_debug.log
          161 kB
          Sean McCarthy
        4. node3.log
          38 kB
          Sean McCarthy
        5. node3_gc.log
          22 kB
          Sean McCarthy
        6. node3_debug.log
          531 kB
          Sean McCarthy
        7. node2.log
          38 kB
          Sean McCarthy
        8. node2_gc.log
          22 kB
          Sean McCarthy
        9. node2_debug.log
          490 kB
          Sean McCarthy
        10. node1.log
          42 kB
          Sean McCarthy
        11. node1_gc.log
          24 kB
          Sean McCarthy
        12. node1_debug.log
          568 kB
          Sean McCarthy
        13. 13058-3.x.patch
          2 kB
          Stefan Podkowinski

        Activity

          People

            spod Stefan Podkowinski
            sean.mccarthy Sean McCarthy
            Stefan Podkowinski
            Paulo Motta
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: