[CASSANDRA-2424] Failed compactation doesn't delete temporary files causing hd to fill

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Fix Version/s: 0.8.1
• Severity: Normal
• Assignee: Unassigned
• Reporter: Thibaut
• Created: 06/Apr/11 09:45
• Updated: 16/Apr/19 09:33
• Resolved: 29/Apr/11 13:30

Description

Hi,

We seem to have a few incorrect keys in one of our columns (Seems to be a memory bit flip). Cassandra will try to compact and compact those tables again but won't delete any temporary files.

INFO [CompactionExecutor:1] 2011-04-05 17:52:45,494 SSTableWriter.java (line 108) Last written key : DecoratedKey(dsearch_1300960988716456<398199492444161_1301855754714_4564397053271578441, 647365617263685f313330303936303938383731363435363c3339383139393439323434343136315f313330313835353735343731345f34353634333937303533323731353738343431)
INFO [CompactionExecutor:1] 2011-04-05 17:52:45,521 SSTableWriter.java (line 109) Current key : DecoratedKey(dsearch_13009609887164564398199492444161_1301855754760_4564395937046043595, 647365617263685f31333030393630393838373136343536343339383139393439323434343136315f313330313835353735343736305f34353634333935393337303436303433353935)
INFO [CompactionExecutor:1] 2011-04-05 17:52:45,521 SSTableWriter.java (line 110) Writing into file /cassandra/data/table_userentries/table_userentries-tmp-f-3550-Data.db
ERROR [CompactionExecutor:1] 2011-04-05 17:52:45,522 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.IOException: Keys must be written in ascending order.
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

ls -l /cassandra/data/table_userentries/table_userentries-tmp*
rw-rr- 1 root root 13G 2011-04-05 16:09 /cassandra/data/table_userentries/table_userentries-tmp-f-3507-Data.db
rw-rr- 1 root root 512M 2011-04-05 16:09 /cassandra/data/table_userentries/table_userentries-tmp-f-3507-Index.db
rw-rr- 1 root root 13G 2011-04-05 16:21 /cassandra/data/table_userentries/table_userentries-tmp-f-3512-Data.db
rw-rr- 1 root root 512M 2011-04-05 16:21 /cassandra/data/table_userentries/table_userentries-tmp-f-3512-Index.db
rw-rr- 1 root root 13G 2011-04-05 16:33 /cassandra/data/table_userentries/table_userentries-tmp-f-3517-Data.db
rw-rr- 1 root root 512M 2011-04-05 16:33 /cassandra/data/table_userentries/table_userentries-tmp-f-3517-Index.db
rw-rr- 1 root root 13G 2011-04-05 16:44 /cassandra/data/table_userentries/table_userentries-tmp-f-3521-Data.db
rw-rr- 1 root root 512M 2011-04-05 16:44 /cassandra/data/table_userentries/table_userentries-tmp-f-3521-Index.db
rw-rr- 1 root root 13G 2011-04-05 16:55 /cassandra/data/table_userentries/table_userentries-tmp-f-3527-Data.db
rw-rr- 1 root root 512M 2011-04-05 16:55 /cassandra/data/table_userentries/table_userentries-tmp-f-3527-Index.db
rw-rr- 1 root root 13G 2011-04-05 17:05 /cassandra/data/table_userentries/table_userentries-tmp-f-3531-Data.db
rw-rr- 1 root root 512M 2011-04-05 17:06 /cassandra/data/table_userentries/table_userentries-tmp-f-3531-Index.db
rw-rr- 1 root root 13G 2011-04-05 17:16 /cassandra/data/table_userentries/table_userentries-tmp-f-3535-Data.db
rw-rr- 1 root root 512M 2011-04-05 17:16 /cassandra/data/table_userentries/table_userentries-tmp-f-3535-Index.db
rw-rr- 1 root root 13G 2011-04-05 17:28 /cassandra/data/table_userentries/table_userentries-tmp-f-3540-Data.db
rw-rr- 1 root root 512M 2011-04-05 17:28 /cassandra/data/table_userentries/table_userentries-tmp-f-3540-Index.db
rw-rr- 1 root root 13G 2011-04-05 17:39 /cassandra/data/table_userentries/table_userentries-tmp-f-3545-Data.db
rw-rr- 1 root root 512M 2011-04-05 17:40 /cassandra/data/table_userentries/table_userentries-tmp-f-3545-Index.db
rw-rr- 1 root root 13G 2011-04-05 17:52 /cassandra/data/table_userentries/table_userentries-tmp-f-3550-Data.db
rw-rr- 1 root root 512M 2011-04-05 17:52 /cassandra/data/table_userentries/table_userentries-tmp-f-3550-Index.db
rw-rr- 1 root root 13G 2011-04-05 18:04 /cassandra/data/table_userentries/table_userentries-tmp-f-3556-Data.db
rw-rr- 1 root root 512M 2011-04-05 18:04 /cassandra/data/table_userentries/table_userentries-tmp-f-3556-Index.db
rw-rr- 1 root root 13G 2011-04-05 18:15 /cassandra/data/table_userentries/table_userentries-tmp-f-3562-Data.db
rw-rr- 1 root root 512M 2011-04-05 18:15 /cassandra/data/table_userentries/table_userentries-tmp-f-3562-Index.db
rw-rr- 1 root root 13G 2011-04-05 18:28 /cassandra/data/table_userentries/table_userentries-tmp-f-3566-Data.db
rw-rr- 1 root root 512M 2011-04-05 18:28 /cassandra/data/table_userentries/table_userentries-tmp-f-3566-Index.db
rw-rr- 1 root root 13G 2011-04-05 18:40 /cassandra/data/table_userentries/table_userentries-tmp-f-3572-Data.db
rw-rr- 1 root root 512M 2011-04-05 18:40 /cassandra/data/table_userentries/table_userentries-tmp-f-3572-Index.db
rw-rr- 1 root root 13G 2011-04-05 18:53 /cassandra/data/table_userentries/table_userentries-tmp-f-3577-Data.db
rw-rr- 1 root root 512M 2011-04-05 18:53 /cassandra/data/table_userentries/table_userentries-tmp-f-3577-Index.db
rw-rr- 1 root root 13G 2011-04-05 19:06 /cassandra/data/table_userentries/table_userentries-tmp-f-3582-Data.db
rw-rr- 1 root root 512M 2011-04-05 19:07 /cassandra/data/table_userentries/table_userentries-tmp-f-3582-Index.db
rw-rr- 1 root root 19G 2011-04-05 19:29 /cassandra/data/table_userentries/table_userentries-tmp-f-3592-Data.db
rw-rr- 1 root root 736M 2011-04-05 19:29 /cassandra/data/table_userentries/table_userentries-tmp-f-3592-Index.db
rw-rr- 1 root root 19G 2011-04-05 19:42 /cassandra/data/table_userentries/table_userentries-tmp-f-3598-Data.db
rw-rr- 1 root root 736M 2011-04-05 19:42 /cassandra/data/table_userentries/table_userentries-tmp-f-3598-Index.db
rw-rr- 1 root root 19G 2011-04-05 19:56 /cassandra/data/table_userentries/table_userentries-tmp-f-3602-Data.db
rw-rr- 1 root root 736M 2011-04-05 19:56 /cassandra/data/table_userentries/table_userentries-tmp-f-3602-Index.db
rw-rr- 1 root root 19G 2011-04-05 20:12 /cassandra/data/table_userentries/table_userentries-tmp-f-3606-Data.db
rw-rr- 1 root root 736M 2011-04-05 20:12 /cassandra/data/table_userentries/table_userentries-tmp-f-3606-Index.db
rw-rr- 1 root root 19G 2011-04-05 20:28 /cassandra/data/table_userentries/table_userentries-tmp-f-3612-Data.db
rw-rr- 1 root root 736M 2011-04-05 20:28 /cassandra/data/table_userentries/table_userentries-tmp-f-3612-Index.db
rw-rr- 1 root root 19G 2011-04-05 20:44 /cassandra/data/table_userentries/table_userentries-tmp-f-3617-Data.db
rw-rr- 1 root root 736M 2011-04-05 20:44 /cassandra/data/table_userentries/table_userentries-tmp-f-3617-Index.db
rw-rr- 1 root root 19G 2011-04-05 21:01 /cassandra/data/table_userentries/table_userentries-tmp-f-3622-Data.db
rw-rr- 1 root root 736M 2011-04-05 21:01 /cassandra/data/table_userentries/table_userentries-tmp-f-3622-Index.db
rw-rr- 1 root root 19G 2011-04-05 21:23 /cassandra/data/table_userentries/table_userentries-tmp-f-3627-Data.db
rw-rr- 1 root root 736M 2011-04-05 21:23 /cassandra/data/table_userentries/table_userentries-tmp-f-3627-Index.db
rw-rr- 1 root root 19G 2011-04-05 21:48 /cassandra/data/table_userentries/table_userentries-tmp-f-3635-Data.db
rw-rr- 1 root root 736M 2011-04-05 21:48 /cassandra/data/table_userentries/table_userentries-tmp-f-3635-Index.db
rw-rr- 1 root root 19G 2011-04-05 22:16 /cassandra/data/table_userentries/table_userentries-tmp-f-3646-Data.db
rw-rr- 1 root root 736M 2011-04-05 22:16 /cassandra/data/table_userentries/table_userentries-tmp-f-3646-Index.db
rw-rr- 1 root root 19G 2011-04-05 22:48 /cassandra/data/table_userentries/table_userentries-tmp-f-3658-Data.db
rw-rr- 1 root root 736M 2011-04-05 22:48 /cassandra/data/table_userentries/table_userentries-tmp-f-3658-Index.db
rw-rr- 1 root root 19G 2011-04-05 23:22 /cassandra/data/table_userentries/table_userentries-tmp-f-3672-Data.db
rw-rr- 1 root root 736M 2011-04-05 23:21 /cassandra/data/table_userentries/table_userentries-tmp-f-3672-Index.db
rw-rr- 1 root root 19G 2011-04-05 23:53 /cassandra/data/table_userentries/table_userentries-tmp-f-3685-Data.db
rw-rr- 1 root root 736M 2011-04-05 23:52 /cassandra/data/table_userentries/table_userentries-tmp-f-3685-Index.db
rw-rr- 1 root root 19G 2011-04-06 00:29 /cassandra/data/table_userentries/table_userentries-tmp-f-3698-Data.db
rw-rr- 1 root root 736M 2011-04-06 00:28 /cassandra/data/table_userentries/table_userentries-tmp-f-3698-Index.db
rw-rr- 1 root root 19G 2011-04-06 01:01 /cassandra/data/table_userentries/table_userentries-tmp-f-3712-Data.db
rw-rr- 1 root root 736M 2011-04-06 01:01 /cassandra/data/table_userentries/table_userentries-tmp-f-3712-Index.db
rw-rr- 1 root root 19G 2011-04-06 01:36 /cassandra/data/table_userentries/table_userentries-tmp-f-3723-Data.db
rw-rr- 1 root root 736M 2011-04-06 01:36 /cassandra/data/table_userentries/table_userentries-tmp-f-3723-Index.db
rw-rr- 1 root root 19G 2011-04-06 02:08 /cassandra/data/table_userentries/table_userentries-tmp-f-3735-Data.db
rw-rr- 1 root root 736M 2011-04-06 02:08 /cassandra/data/table_userentries/table_userentries-tmp-f-3735-Index.db
rw-rr- 1 root root 13G 2011-04-06 02:32 /cassandra/data/table_userentries/table_userentries-tmp-f-3742-Data.db
rw-rr- 1 root root 512M 2011-04-06 02:32 /cassandra/data/table_userentries/table_userentries-tmp-f-3742-Index.db
rw-rr- 1 root root 4.5G 2011-04-06 02:42 /cassandra/data/table_userentries/table_userentries-tmp-f-3748-Data.db
rw-rr- 1 root root 168M 2011-04-06 02:42 /cassandra/data/table_userentries/table_userentries-tmp-f-3748-Index.db