Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-731

a few insert operations failed while bootstrapping

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 0.5
    • Component/s: None
    • Labels:
      None
    • Severity:
      Normal

      Description

      I inserted 10000 key/value while bootstrapping and found 2 insert operations failed.

      DEBUG [pool-1-thread-63] 2010-01-20 17:01:57,033 StorageProxy.java (line 225) insert writing key 15530 to 10981@/10.81.37.65
      ERROR [pool-1-thread-46] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
      java.lang.AssertionError
      at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
      at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
      at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
      at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
      at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
      at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
      at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
      at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
      at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java.lang.Thread.run(Thread.java:619)
      ERROR [pool-1-thread-44] 2010-01-20 17:01:57,033 Cassandra.java (line 1064) Internal error processing insert
      java.lang.AssertionError
      at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:157)
      at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
      at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1178)
      at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
      at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
      at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
      at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
      at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
      at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java.lang.Thread.run(Thread.java:619)

      I traced the code and found the following assertion failed :
      /* org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(Collection<InetAddress>) */
      assert map.size() == targets.size();

      The following reasons caused this issue:
      1) targets is a list , not a map, as a result there may be some duplicated IP.
      2) The following codes are not atomic :
      org.apache.cassandra.service.StorageService.handleStateNormal(InetAddress, String)
      tokenMetadata_.updateNormalToken(token, endPoint);
      calculatePendingRanges();

      That's to say the IP may be both in the naturalEndpoints and pendingRanges.

      eg :
      targets is IPa, IPb, IPc, IPa; (size = 4)
      then, the map will be IPa, IPb, IPc. (size = 3)
      as a result, assert failed.

        Attachments

          Activity

            People

            • Assignee:
              jaakko Jaakko Laine
              Reporter:
              david.pan david.pan
              Authors:
              Jaakko Laine
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified