Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7145

FileNotFoundException during compaction

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 1.2.19, 2.0.11, 2.1.0
    • Component/s: Local/Compaction
    • Labels:
      None
    • Environment:

      CentOS 6.3, Datastax Enterprise 4.0.1 (Cassandra 2.0.5), Java 1.7.0_55

    • Severity:
      Normal

      Description

      I can't finish any compaction because my nodes always throw a "FileNotFoundException". I've already tried the following but nothing helped:

      1. nodetool flush
      2. nodetool repair (ends with RuntimeException; see attachment)
      3. node restart (via dse cassandra-stop)

      Whenever I restart the nodes, another type of exception is logged (see attachment) somewhere near the end of startup process. This particular exception doesn't seem to be critical because the nodes still manage to finish the startup and become online.

      I don't have specific steps to reproduce the problem that I'm experiencing with compaction and repair. I'm in the middle of migrating 4.8 billion rows from MySQL via SSTableLoader.

      Some things that may or may not be relevant:
      1. I didn't drop and recreate the keyspace (so probably not related to CASSANDRA-4857)
      2. I do the bulk-loading in batches of 1 to 20 millions rows. When a batch reaches 100% total progress (i.e. starts to build secondary index), I kill the sstableloader process and cancel the index build
      3. I restart the nodes occasionally. It's possible that there is an on-going compaction during one of those restarts.

      Related StackOverflow question (mine): http://stackoverflow.com/questions/23435847/filenotfoundexception-during-compaction

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              marcuse Marcus Eriksson Assign to me
              Reporter:
              pjc08 PJ
              Authors:
              Marcus Eriksson
              Reviewers:
              Yuki Morishita

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment