Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-713

Stacktrace when node taken offline

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Cannot Reproduce
    • Fix Version/s: 0.5
    • Component/s: None
    • Labels:
      None
    • Severity:
      Normal

      Description

      I took a node offline last week and then attempted to re-bootstrap its token range with a new cassandra install on the same IP. I made gossip forget about the node by restarting all other instances, then brought up the new node. It said was bootstrapping, but it never finished bootstrapping after several days. The node never showed up in the ring, but when I take it offline, I get the following exception continually from all other nodes in the cluster:

      ERROR [pool-1-thread-8] 2010-01-18 21:01:32,405 Cassandra.java (line 1096) Internal error processing batch_insert
      java.lang.NullPointerException
      at org.apache.cassandra.dht.BigIntegerToken.compareTo(BigIntegerToken.java:38)
      at org.apache.cassandra.dht.BigIntegerToken.compareTo(BigIntegerToken.java:23)
      at java.util.Collections.indexedBinarySearch(Collections.java:215)
      at java.util.Collections.binarySearch(Collections.java:201)
      at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:130)
      at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
      at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1183)
      at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
      at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
      at org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:445)
      at org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:1088)
      at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
      at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)

      In addition, I get frequent UnavailableExceptions on the other nodes.

      I cannot remove the token range for this node because it never officially joined the ring.

        Attachments

          Activity

            People

            • Assignee:
              jaakko Jaakko Laine
              Reporter:
              ryandaum Ryan Daum
              Authors:
              Jaakko Laine
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: