Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-6008

Getting 'This should never happen' error at startup due to sstables missing

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Duplicate
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Severity:
      Normal

      Description

      Exception encountered during startup: "Unfinished compactions reference missing sstables. This should never happen since compactions are marked finished before we start removing the old sstables"

      This happens when sstables that have been compacted away are removed, but they still have entries in the system.compactions_in_progress table.

      Normally this should not happen because the entries in system.compactions_in_progress are deleted before the old sstables are deleted.

      However at startup recovery time, old sstables are deleted (NOT BEFORE they are removed from the compactions_in_progress table) and then after that is done it does a truncate using SystemKeyspace.discardCompactionsInProgress

      We ran into a case where the disk filled up and the node died and was bounced and then failed to truncate this table on startup, and then got stuck hitting this exception in ColumnFamilyStore.removeUnfinishedCompactionLeftovers.

      Maybe on startup we can delete from this table incrementally as we clean stuff up in the same way that compactions delete from this table before they delete old sstables.

        Attachments

        1. 6008-trunk-v1.patch
          35 kB
          Tom Hobbs
        2. 6008-2.0-v1.patch
          33 kB
          Tom Hobbs
        3. 6008-2.0-part2.patch
          18 kB
          Tom Hobbs

        Issue Links

          Activity

            People

            • Assignee:
              thobbs Tom Hobbs Assign to me
              Reporter:
              johnyoh John Carrino
              Authors:
              Tom Hobbs
              Reviewers:
              Jonathan Ellis

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment