Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-2424

Failed compactation doesn't delete temporary files causing hd to fill

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Duplicate
    • Fix Version/s: 0.8.1
    • Component/s: None
    • Labels:
      None
    • Severity:
      Normal

      Description

      Hi,

      We seem to have a few incorrect keys in one of our columns (Seems to be a memory bit flip). Cassandra will try to compact and compact those tables again but won't delete any temporary files.

      INFO [CompactionExecutor:1] 2011-04-05 17:52:45,494 SSTableWriter.java (line 108) Last written key : DecoratedKey(dsearch_1300960988716456<398199492444161_1301855754714_4564397053271578441, 647365617263685f313330303936303938383731363435363c3339383139393439323434343136315f313330313835353735343731345f34353634333937303533323731353738343431)
      INFO [CompactionExecutor:1] 2011-04-05 17:52:45,521 SSTableWriter.java (line 109) Current key : DecoratedKey(dsearch_13009609887164564398199492444161_1301855754760_4564395937046043595, 647365617263685f31333030393630393838373136343536343339383139393439323434343136315f313330313835353735343736305f34353634333935393337303436303433353935)
      INFO [CompactionExecutor:1] 2011-04-05 17:52:45,521 SSTableWriter.java (line 110) Writing into file /cassandra/data/table_userentries/table_userentries-tmp-f-3550-Data.db
      ERROR [CompactionExecutor:1] 2011-04-05 17:52:45,522 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
      java.io.IOException: Keys must be written in ascending order.
      at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
      at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
      at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
      at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
      at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

      ls -l /cassandra/data/table_userentries/table_userentries-tmp*
      rw-rr- 1 root root 13G 2011-04-05 16:09 /cassandra/data/table_userentries/table_userentries-tmp-f-3507-Data.db
      rw-rr- 1 root root 512M 2011-04-05 16:09 /cassandra/data/table_userentries/table_userentries-tmp-f-3507-Index.db
      rw-rr- 1 root root 13G 2011-04-05 16:21 /cassandra/data/table_userentries/table_userentries-tmp-f-3512-Data.db
      rw-rr- 1 root root 512M 2011-04-05 16:21 /cassandra/data/table_userentries/table_userentries-tmp-f-3512-Index.db
      rw-rr- 1 root root 13G 2011-04-05 16:33 /cassandra/data/table_userentries/table_userentries-tmp-f-3517-Data.db
      rw-rr- 1 root root 512M 2011-04-05 16:33 /cassandra/data/table_userentries/table_userentries-tmp-f-3517-Index.db
      rw-rr- 1 root root 13G 2011-04-05 16:44 /cassandra/data/table_userentries/table_userentries-tmp-f-3521-Data.db
      rw-rr- 1 root root 512M 2011-04-05 16:44 /cassandra/data/table_userentries/table_userentries-tmp-f-3521-Index.db
      rw-rr- 1 root root 13G 2011-04-05 16:55 /cassandra/data/table_userentries/table_userentries-tmp-f-3527-Data.db
      rw-rr- 1 root root 512M 2011-04-05 16:55 /cassandra/data/table_userentries/table_userentries-tmp-f-3527-Index.db
      rw-rr- 1 root root 13G 2011-04-05 17:05 /cassandra/data/table_userentries/table_userentries-tmp-f-3531-Data.db
      rw-rr- 1 root root 512M 2011-04-05 17:06 /cassandra/data/table_userentries/table_userentries-tmp-f-3531-Index.db
      rw-rr- 1 root root 13G 2011-04-05 17:16 /cassandra/data/table_userentries/table_userentries-tmp-f-3535-Data.db
      rw-rr- 1 root root 512M 2011-04-05 17:16 /cassandra/data/table_userentries/table_userentries-tmp-f-3535-Index.db
      rw-rr- 1 root root 13G 2011-04-05 17:28 /cassandra/data/table_userentries/table_userentries-tmp-f-3540-Data.db
      rw-rr- 1 root root 512M 2011-04-05 17:28 /cassandra/data/table_userentries/table_userentries-tmp-f-3540-Index.db
      rw-rr- 1 root root 13G 2011-04-05 17:39 /cassandra/data/table_userentries/table_userentries-tmp-f-3545-Data.db
      rw-rr- 1 root root 512M 2011-04-05 17:40 /cassandra/data/table_userentries/table_userentries-tmp-f-3545-Index.db
      rw-rr- 1 root root 13G 2011-04-05 17:52 /cassandra/data/table_userentries/table_userentries-tmp-f-3550-Data.db
      rw-rr- 1 root root 512M 2011-04-05 17:52 /cassandra/data/table_userentries/table_userentries-tmp-f-3550-Index.db
      rw-rr- 1 root root 13G 2011-04-05 18:04 /cassandra/data/table_userentries/table_userentries-tmp-f-3556-Data.db
      rw-rr- 1 root root 512M 2011-04-05 18:04 /cassandra/data/table_userentries/table_userentries-tmp-f-3556-Index.db
      rw-rr- 1 root root 13G 2011-04-05 18:15 /cassandra/data/table_userentries/table_userentries-tmp-f-3562-Data.db
      rw-rr- 1 root root 512M 2011-04-05 18:15 /cassandra/data/table_userentries/table_userentries-tmp-f-3562-Index.db
      rw-rr- 1 root root 13G 2011-04-05 18:28 /cassandra/data/table_userentries/table_userentries-tmp-f-3566-Data.db
      rw-rr- 1 root root 512M 2011-04-05 18:28 /cassandra/data/table_userentries/table_userentries-tmp-f-3566-Index.db
      rw-rr- 1 root root 13G 2011-04-05 18:40 /cassandra/data/table_userentries/table_userentries-tmp-f-3572-Data.db
      rw-rr- 1 root root 512M 2011-04-05 18:40 /cassandra/data/table_userentries/table_userentries-tmp-f-3572-Index.db
      rw-rr- 1 root root 13G 2011-04-05 18:53 /cassandra/data/table_userentries/table_userentries-tmp-f-3577-Data.db
      rw-rr- 1 root root 512M 2011-04-05 18:53 /cassandra/data/table_userentries/table_userentries-tmp-f-3577-Index.db
      rw-rr- 1 root root 13G 2011-04-05 19:06 /cassandra/data/table_userentries/table_userentries-tmp-f-3582-Data.db
      rw-rr- 1 root root 512M 2011-04-05 19:07 /cassandra/data/table_userentries/table_userentries-tmp-f-3582-Index.db
      rw-rr- 1 root root 19G 2011-04-05 19:29 /cassandra/data/table_userentries/table_userentries-tmp-f-3592-Data.db
      rw-rr- 1 root root 736M 2011-04-05 19:29 /cassandra/data/table_userentries/table_userentries-tmp-f-3592-Index.db
      rw-rr- 1 root root 19G 2011-04-05 19:42 /cassandra/data/table_userentries/table_userentries-tmp-f-3598-Data.db
      rw-rr- 1 root root 736M 2011-04-05 19:42 /cassandra/data/table_userentries/table_userentries-tmp-f-3598-Index.db
      rw-rr- 1 root root 19G 2011-04-05 19:56 /cassandra/data/table_userentries/table_userentries-tmp-f-3602-Data.db
      rw-rr- 1 root root 736M 2011-04-05 19:56 /cassandra/data/table_userentries/table_userentries-tmp-f-3602-Index.db
      rw-rr- 1 root root 19G 2011-04-05 20:12 /cassandra/data/table_userentries/table_userentries-tmp-f-3606-Data.db
      rw-rr- 1 root root 736M 2011-04-05 20:12 /cassandra/data/table_userentries/table_userentries-tmp-f-3606-Index.db
      rw-rr- 1 root root 19G 2011-04-05 20:28 /cassandra/data/table_userentries/table_userentries-tmp-f-3612-Data.db
      rw-rr- 1 root root 736M 2011-04-05 20:28 /cassandra/data/table_userentries/table_userentries-tmp-f-3612-Index.db
      rw-rr- 1 root root 19G 2011-04-05 20:44 /cassandra/data/table_userentries/table_userentries-tmp-f-3617-Data.db
      rw-rr- 1 root root 736M 2011-04-05 20:44 /cassandra/data/table_userentries/table_userentries-tmp-f-3617-Index.db
      rw-rr- 1 root root 19G 2011-04-05 21:01 /cassandra/data/table_userentries/table_userentries-tmp-f-3622-Data.db
      rw-rr- 1 root root 736M 2011-04-05 21:01 /cassandra/data/table_userentries/table_userentries-tmp-f-3622-Index.db
      rw-rr- 1 root root 19G 2011-04-05 21:23 /cassandra/data/table_userentries/table_userentries-tmp-f-3627-Data.db
      rw-rr- 1 root root 736M 2011-04-05 21:23 /cassandra/data/table_userentries/table_userentries-tmp-f-3627-Index.db
      rw-rr- 1 root root 19G 2011-04-05 21:48 /cassandra/data/table_userentries/table_userentries-tmp-f-3635-Data.db
      rw-rr- 1 root root 736M 2011-04-05 21:48 /cassandra/data/table_userentries/table_userentries-tmp-f-3635-Index.db
      rw-rr- 1 root root 19G 2011-04-05 22:16 /cassandra/data/table_userentries/table_userentries-tmp-f-3646-Data.db
      rw-rr- 1 root root 736M 2011-04-05 22:16 /cassandra/data/table_userentries/table_userentries-tmp-f-3646-Index.db
      rw-rr- 1 root root 19G 2011-04-05 22:48 /cassandra/data/table_userentries/table_userentries-tmp-f-3658-Data.db
      rw-rr- 1 root root 736M 2011-04-05 22:48 /cassandra/data/table_userentries/table_userentries-tmp-f-3658-Index.db
      rw-rr- 1 root root 19G 2011-04-05 23:22 /cassandra/data/table_userentries/table_userentries-tmp-f-3672-Data.db
      rw-rr- 1 root root 736M 2011-04-05 23:21 /cassandra/data/table_userentries/table_userentries-tmp-f-3672-Index.db
      rw-rr- 1 root root 19G 2011-04-05 23:53 /cassandra/data/table_userentries/table_userentries-tmp-f-3685-Data.db
      rw-rr- 1 root root 736M 2011-04-05 23:52 /cassandra/data/table_userentries/table_userentries-tmp-f-3685-Index.db
      rw-rr- 1 root root 19G 2011-04-06 00:29 /cassandra/data/table_userentries/table_userentries-tmp-f-3698-Data.db
      rw-rr- 1 root root 736M 2011-04-06 00:28 /cassandra/data/table_userentries/table_userentries-tmp-f-3698-Index.db
      rw-rr- 1 root root 19G 2011-04-06 01:01 /cassandra/data/table_userentries/table_userentries-tmp-f-3712-Data.db
      rw-rr- 1 root root 736M 2011-04-06 01:01 /cassandra/data/table_userentries/table_userentries-tmp-f-3712-Index.db
      rw-rr- 1 root root 19G 2011-04-06 01:36 /cassandra/data/table_userentries/table_userentries-tmp-f-3723-Data.db
      rw-rr- 1 root root 736M 2011-04-06 01:36 /cassandra/data/table_userentries/table_userentries-tmp-f-3723-Index.db
      rw-rr- 1 root root 19G 2011-04-06 02:08 /cassandra/data/table_userentries/table_userentries-tmp-f-3735-Data.db
      rw-rr- 1 root root 736M 2011-04-06 02:08 /cassandra/data/table_userentries/table_userentries-tmp-f-3735-Index.db
      rw-rr- 1 root root 13G 2011-04-06 02:32 /cassandra/data/table_userentries/table_userentries-tmp-f-3742-Data.db
      rw-rr- 1 root root 512M 2011-04-06 02:32 /cassandra/data/table_userentries/table_userentries-tmp-f-3742-Index.db
      rw-rr- 1 root root 4.5G 2011-04-06 02:42 /cassandra/data/table_userentries/table_userentries-tmp-f-3748-Data.db
      rw-rr- 1 root root 168M 2011-04-06 02:42 /cassandra/data/table_userentries/table_userentries-tmp-f-3748-Index.db

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tbritz Thibaut
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: