Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-8623

sstablesplit fails *randomly* with Data component is missing

    XMLWordPrintableJSON

    Details

    • Severity:
      Normal

      Description

      I'm experiencing an issue related to sstablesplit. I would like to understand if I am doing something wrong or there is an issue in the split process. The process fails randomly with the following exception:

      ERROR 02:17:36 Error in ThreadPoolExecutor
      java.lang.AssertionError: Data component is missing for sstable./tools/bin/../../data/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-16
      

      See attached output.log file. The process never stops after this exception and I've also seen the dataset growing indefinitely (number of sstables).

      • I have not been able to reproduce the issue with a single sstablesplit command. ie, specifying all files with glob matching.
      • I can reproduce the bug if I call multiple sstablesplit one file at the time (the way ccm does)

      Here is the test case file to reproduce the bug:

      https://drive.google.com/file/d/0BwZ_GPM33j6KdVh0NTdkOWV2R1E/view?usp=sharing

      1. Download the split_issue.tar.gz file. It includes latest cassandra-2.1 branch binaries.
      2. Extract it
      3. CD inside the use case directory
      4. Download the dataset (2G) just to be sure we have the same thing, and place it in the working directory.
      https://docs.google.com/uc?id=0BwZ_GPM33j6KV3ViNnpPcVFndUU&export=download
      5. The first time, run ./test.sh. This will setup and run a test.
      6. The next times, you can only run ./test --no-setup . This will only reset the dataset as its initial state and re-run the test. You might have to run the tests some times before experiencing it... but I'm always able with only 2-3 runs.

        Attachments

        1. 0001-make-sure-we-finish-compactions-before-waiting-for-d.patch
          6 kB
          Marcus Eriksson
        2. 8623-v2.patch
          9 kB
          Marcus Eriksson
        3. output.log
          4 kB
          Alan Boudreault
        4. output2.log
          1.34 MB
          Alan Boudreault

          Activity

            People

            • Assignee:
              marcuse Marcus Eriksson
              Reporter:
              aboudreault Alan Boudreault
              Authors:
              Marcus Eriksson
              Reviewers:
              Yuki Morishita
              Tester:
              Alan Boudreault
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: