Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Invalid
    • Fix Version/s: None
    • Labels:
      None
    • Environment:

      oracle java 1.6u35, oracle java 1.6u37, linux 2.6.32

      Description

      ERROR [ValidationExecutor:3] 2012-10-24 16:41:24,310 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[ValidationExecutor:3,1,main]
      java.lang.AssertionError: row DecoratedKey(73897609397945816944009475942793437831, 37663364336664642d353532642d343736392d613437662d376339656339623637386138) received out of order wrt DecoratedKey(104298997089014216307657576199722210661,
      65396135326632372d343065352d343864372d616362372d326665373738616434366334)
      at org.apache.cassandra.service.AntiEntropyService$Validator.add(AntiEntropyService.java:349)
      at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:716)
      at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:69)
      at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:442)
      at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)

      I've also found in log file:
      ERROR [CompactionExecutor:2974] 2012-10-26 02:59:21,384 LeveledManifest.java (line 223) At level 3, SSTableReader(path='/var/lib/cassandra/data/Archive/Messages/Archive-Messages-hf-97756-Data.db') [DecoratedKey(72089347517688459378407444327185359483, 31343563626135612d323439342d346633312d616130382d653133623336303032363464), DecoratedKey(104298997089014216307657576199722210661, 65396135326632372d343065352d343864372d616362372d326665373738616434366334)] overlaps SSTableReader(path='/var/lib/cassandra/data/Archive/Messages/Archive-Messages-hf-97519-Data.db') [DecoratedKey(73897609397945816944009475942793437831, 37663364336664642d353532642d343736392d613437662d376339656339623637386138), DecoratedKey(104617537605721927860229960625668693343, 39613736316364352d313361392d343163322d383361642d623635366163623032376430)]. This is caused by a bug in Cassandra 1.1.0 .. 1.1.3. Sending back to L0. If you have not yet run scrub, you should do so since you may also have rows out-of-order within an sstable

      but I'd have run scrub on all nodes before repair action

        Activity

        Hide
        Brandon Williams added a comment -

        You can do it in a rolling fashion (your latter description.)

        Show
        Brandon Williams added a comment - You can do it in a rolling fashion (your latter description.)
        Hide
        Zenek Kraweznik added a comment -

        I can't find enough information about it in documentation.

        Should I stop cassandra on every node at the same time and run sstablescrub? Should I stop cassandra on one node, run sstablecrub, start cassandra and stop it on next node?

        Show
        Zenek Kraweznik added a comment - I can't find enough information about it in documentation. Should I stop cassandra on every node at the same time and run sstablescrub? Should I stop cassandra on one node, run sstablecrub, start cassandra and stop it on next node?
        Hide
        Jonathan Ellis added a comment -

        You need to run offline scrub to fix overlapping-sstables-within-a-level.

        Show
        Jonathan Ellis added a comment - You need to run offline scrub to fix overlapping-sstables-within-a-level.

          People

          • Assignee:
            Unassigned
            Reporter:
            Zenek Kraweznik
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development