Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7892

Hashmap IllegalStateException in anticompaction

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 2.1.0, 2.1.1
    • None
    • None
    • Cassandra 2.1 rc 5

    • Normal

    Description

      ERROR [AntiEntropyStage:4] 2014-09-05 19:55:06,439 CassandraDaemon.java:166 - Exception in thread Thread[AntiEntropyStage:4,5,main]
      java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException
              at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:117) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
              at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_60]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_60]
              at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
      Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException
              at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
              at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
              at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:113) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
              ... 4 common frames omitted
      Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException
              at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.7.0_60]
              at java.util.concurrent.FutureTask.get(FutureTask.java:188) ~[na:1.7.0_60]
              at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
              ... 6 common frames omitted
      Caused by: java.lang.IllegalStateException: null
              at java.util.HashMap$HashIterator.remove(HashMap.java:938) ~[na:1.7.0_60]
              at org.apache.cassandra.db.compaction.CompactionManager.performAnticompaction(CompactionManager.java:425) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
              at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:381) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
              at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_60]
              at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60]
              ... 3 common frames omitted
      

      And on a different host:

      ERROR [CompactionExecutor:43] 2014-09-05 19:55:06,376 CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:4
      3,1,main]
      java.lang.IllegalStateException: null
              at java.util.HashMap$HashIterator.remove(HashMap.java:938) ~[na:1.7.0_60]
              at org.apache.cassandra.db.compaction.CompactionManager.performAnticompaction(CompactionManager.java:425) ~[apache-cassa
      ndra-2.1.0-rc5.jar:2.1.0-rc5]
              at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:381) ~[apache-cassandra-2.1
      .0-rc5.jar:2.1.0-rc5]
              at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_60]
              at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_60]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
              at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
      

      The exceptions happened in our test environment after I added a bunch of new-nodes at the same time that was in a different (previously non-existing) datacenter. The datacenter is determined using GossipingPropertyFileSnitch and started repairs on a bunch of nodes at the same time using

      nodetool -h <node> repair -inc -par
      

      The nodes in the old datacenter had previously been repaired using the -inc option.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            mishail Mikhail Stepura Assign to me
            phb Johan Bjork
            Mikhail Stepura
            Yuki Morishita
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment