Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-10292

java.lang.AssertionError: attempted to delete non-existing file CommitLog...

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Urgent
    • Resolution: Fixed
    • None
    • None
    • None
    • CentOS Linux 7.1.1503, Cassandra 2.1.8 stable version, 6 nodes cluster

    • Critical

    Description

      From time to time some nodes are stopping to work due to error in logs like this:
      INFO [CompactionExecutor:2475] 2015-09-09 12:36:50,363 CompactionTask.java:274 - Compacted 4 sstables to [/mnt/cassandra--storage-machine/data/system/compactions_in_progress-55080ab05d9c38
      8690a4acb25fe1f77b/system-compactions_in_progress-ka-126,]. 419 bytes to 42 (~10% of original) in 33ms = 0.001214MB/s. 4 total partitions merged to 1. Partition merge counts were

      {2:2, }

      INFO [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:34,166 ColumnFamilyStore.java:912 - Enqueuing flush of settings: 78364 (0%) on-heap, 0 (0%) off-heap
      INFO [MemtableFlushWriter:301] 2015-09-09 12:52:34,172 Memtable.java:347 - Writing Memtable-settings@1126939979(0.113KiB serialized bytes, 1850 ops, 0%/0% of on/off-heap limit)
      INFO [MemtableFlushWriter:301] 2015-09-09 12:52:34,174 Memtable.java:382 - Completed flushing /mnt/cassandra--storage-machine/data/OpsCenter/settings-464866c04b1311e590698d1a9fd4ba8b/OpsCe
      nter-settings-tmp-ka-12-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1441362636571, position=33554415)
      ERROR [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:34,194 StorageService.java:453 - Stopping gossiper
      WARN [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:34,195 StorageService.java:359 - Stopping gossip by operator request
      INFO [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:34,195 Gossiper.java:1410 - Announcing shutdown
      ERROR [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:36,195 StorageService.java:458 - Stopping RPC server
      INFO [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:36,196 ThriftServer.java:142 - Stop listening to thrift clients
      ERROR [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:36,204 StorageService.java:463 - Stopping native transport
      INFO [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:36,422 Server.java:213 - Stop listening for CQL clients
      ERROR [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:36,423 CommitLog.java:397 - Failed managing commit log segments. Commit disk failure policy is stop; terminating thread
      java.lang.AssertionError: attempted to delete non-existing file CommitLog-4-1441362636316.log
      at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:126) ~[apache-cassandra-2.1.8.jar:2.1.8]
      at org.apache.cassandra.db.commitlog.CommitLogSegment.delete(CommitLogSegment.java:343) ~[apache-cassandra-2.1.8.jar:2.1.8]
      at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$5.call(CommitLogSegmentManager.java:418) ~[apache-cassandra-2.1.8.jar:2.1.8]
      at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$5.call(CommitLogSegmentManager.java:413) ~[apache-cassandra-2.1.8.jar:2.1.8]
      at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:152) ~[apache-cassandra-2.1.8.jar:2.1.8]
      at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [apache-cassandra-2.1.8.jar:2.1.8]
      at java.lang.Thread.run(Thread.java:745) [na:1.7.0_85]

      After I create missing commit log file and restart cassandra service everything is OK then.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            dejw Dawid Szejnfeld
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment