Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 2.0 beta 1
    • Component/s: Core
    • Labels:

      Description

      LazilyCompactedRow reads all data twice to compact a row which is obviously inefficient. The main reason we do that is to compute the row header. However, CASSANDRA-2319 have removed the main part of that row header. What remains is the size in bytes and the number of columns, but it should be relatively simple to remove those, which would then remove the need for the two-phase compaction.

      1. scrub-error.txt
        20 kB
        Jason Brown

        Issue Links

          Activity

          Hide
          Sylvain Lebresne added a comment -

          I'll note that an initial idea could be to keep the row header as it is (post CASSANDRA-2319), and during compaction to keep the space for the row size and column count, compact all columns, and seek back to write those two values. However, compression forbids us to do that, so we'll have to really remove those part two. However, we can trade the column count by writing a specific marker to mark the end of a row. As for the data size, we can get it from the index.

          Show
          Sylvain Lebresne added a comment - I'll note that an initial idea could be to keep the row header as it is (post CASSANDRA-2319 ), and during compaction to keep the space for the row size and column count, compact all columns, and seek back to write those two values. However, compression forbids us to do that, so we'll have to really remove those part two. However, we can trade the column count by writing a specific marker to mark the end of a row. As for the data size, we can get it from the index.
          Hide
          Jonathan Ellis added a comment -

          This will be a big win, I'm excited.

          Show
          Jonathan Ellis added a comment - This will be a big win, I'm excited.
          Hide
          MaHaiyang added a comment -

          About a years ago ,I tried to do this ,but find it difficult .

          Show
          MaHaiyang added a comment - About a years ago ,I tried to do this ,but find it difficult .
          Hide
          Jonathan Ellis added a comment -

          Removing the row-level bloom filter would make this a lot simpler.

          Show
          Jonathan Ellis added a comment - Removing the row-level bloom filter would make this a lot simpler.
          Hide
          Sylvain Lebresne added a comment -

          Removing the row-level bloom filter would make this a lot simpler.

          Note that it really matters at the end of the day, but just to make sure we're on the same page, now that the row-level filters have been promoted to the index file, I don't think removing or keeping will have much impact on this ticket.

          Show
          Sylvain Lebresne added a comment - Removing the row-level bloom filter would make this a lot simpler. Note that it really matters at the end of the day, but just to make sure we're on the same page, now that the row-level filters have been promoted to the index file, I don't think removing or keeping will have much impact on this ticket.
          Hide
          Jason Brown added a comment -

          Sylvain Lebresne I think you are right about that, and disambiguating the two tickets (at least, in my brain) should make this a little easier.

          Show
          Jason Brown added a comment - Sylvain Lebresne I think you are right about that, and disambiguating the two tickets (at least, in my brain) should make this a little easier.
          Hide
          Jonathan Ellis added a comment -

          Ugh, this is almost straightforward. But doing without column count is kind of a bitch. Fortunately we can use 00/00 as the end-of-row marker since empty column names are not allowed.

          Show
          Jonathan Ellis added a comment - Ugh, this is almost straightforward. But doing without column count is kind of a bitch. Fortunately we can use 00/00 as the end-of-row marker since empty column names are not allowed.
          Hide
          Jason Brown added a comment -

          Removing the column column and row size and using an end-of-row (EOR) marker seems straightforward, but the devil has been in the details. The main challenge is reading to you read columns until the EOR marker, and you return false on the next hasNext(). The problem is if a higher-level iterator calls hasNext() again on that row, you've already moved the the file pointer past the EOR row (and are at the beginning of the next row). So you look for the EOR marker, which you've already read for that row, but the pointer reads either the key for the next row or EOF - either way, not so good. It took me while to work though that one, and I think I'm done resolving the bugs around removing column count.

          Removing the row size has been largely easier, but I'm working through (hopefully) the last issue on SSTS.KeyScanningIterator. It uses row size to calculate the start of the next row. Without that, if you skip a row based solely upon key, then I need to efficiently advance the file pointer the the head of the next row.

          Hoping to finish very soon.

          Show
          Jason Brown added a comment - Removing the column column and row size and using an end-of-row (EOR) marker seems straightforward, but the devil has been in the details. The main challenge is reading to you read columns until the EOR marker, and you return false on the next hasNext(). The problem is if a higher-level iterator calls hasNext() again on that row, you've already moved the the file pointer past the EOR row (and are at the beginning of the next row). So you look for the EOR marker, which you've already read for that row, but the pointer reads either the key for the next row or EOF - either way, not so good. It took me while to work though that one, and I think I'm done resolving the bugs around removing column count. Removing the row size has been largely easier, but I'm working through (hopefully) the last issue on SSTS.KeyScanningIterator. It uses row size to calculate the start of the next row. Without that, if you skip a row based solely upon key, then I need to efficiently advance the file pointer the the head of the next row. Hoping to finish very soon.
          Hide
          Jonathan Ellis added a comment - - edited

          Sorry, I got caught up in this over the weekend since the more I dug on CASSANDRA-5344 the more it looked like I was actually solving this first. I've pushed my first draft to http://github.com/jbellis/cassandra/tree/4180.

          I see DefsTest fail occasionally but it is not 100% reproducible, so I'm not sure if it was caused by my changes. I also see CFSTest log errors sometimes but that one is definitely not related (CASSANDRA-5410).

          (Testing was a bitch at first since sstable errors would just cause the schema loader to break violently; hence the option I added to just inject the schema directly, without going through the migration path. That allowed enough tests to run to track down the problems.)

          Everything was fairly straightforward except SSTableScanner. (Sounds like the same thing Jason ran into.) I simplified things by noting that seekTo was only used to initialize the scanner to a certain starting point, so I pulled that into the constructor to make seeking mid-iteration a non-concern. (This also allowed removing SSTableBoundedScanner.) I also merged KeyScanningIterator and FilteringKSI; FKSI already had most of the code needed to compute data size from the index entries, which compaction needed to decide whether to use an eager or lazy approach.

          There were a lot of places that just one-off sstable reading that were easy to miss. This smells fishy to me but it wasn't obvious how to re-organize things to make it unnecessary, so I haven't tried to solve that here.

          I also haven't tried to update scrub for the new format.

          Show
          Jonathan Ellis added a comment - - edited Sorry, I got caught up in this over the weekend since the more I dug on CASSANDRA-5344 the more it looked like I was actually solving this first. I've pushed my first draft to http://github.com/jbellis/cassandra/tree/4180 . I see DefsTest fail occasionally but it is not 100% reproducible, so I'm not sure if it was caused by my changes. I also see CFSTest log errors sometimes but that one is definitely not related ( CASSANDRA-5410 ). (Testing was a bitch at first since sstable errors would just cause the schema loader to break violently; hence the option I added to just inject the schema directly, without going through the migration path. That allowed enough tests to run to track down the problems.) Everything was fairly straightforward except SSTableScanner. (Sounds like the same thing Jason ran into.) I simplified things by noting that seekTo was only used to initialize the scanner to a certain starting point, so I pulled that into the constructor to make seeking mid-iteration a non-concern. (This also allowed removing SSTableBoundedScanner.) I also merged KeyScanningIterator and FilteringKSI; FKSI already had most of the code needed to compute data size from the index entries, which compaction needed to decide whether to use an eager or lazy approach. There were a lot of places that just one-off sstable reading that were easy to miss. This smells fishy to me but it wasn't obvious how to re-organize things to make it unnecessary, so I haven't tried to solve that here. I also haven't tried to update scrub for the new format.
          Hide
          Jason Brown added a comment -

          On the whole, LGTM.

          I think Jonathan and I had the same basic notion of how to implement this, and we ran into many of the same problems (schema loading borked hard and fast). Admittedly, the one-off sstable reads were what kept kicking me in the shins (repeatedly), and what kept taking me down wrong paths. I agree about the fishy smell with knowledge of the SSTable format spread around, but minimally that's for a another ticket.

          We can probably remove the

          output != null

          check in ColumnIndex.Builder.add() as LCR is no longer passing in null (on the first pass).

          Jonathan mentioned he didn't try to scrub, but I just tried it out locally, and it failed with an OOM error. I'd look into it more, but brain power running out this late at night. I'm attaching a file with the scrub errors. Will look into it tomorrow.

          Show
          Jason Brown added a comment - On the whole, LGTM. I think Jonathan and I had the same basic notion of how to implement this, and we ran into many of the same problems (schema loading borked hard and fast). Admittedly, the one-off sstable reads were what kept kicking me in the shins (repeatedly), and what kept taking me down wrong paths. I agree about the fishy smell with knowledge of the SSTable format spread around, but minimally that's for a another ticket. We can probably remove the output != null check in ColumnIndex.Builder.add() as LCR is no longer passing in null (on the first pass). Jonathan mentioned he didn't try to scrub, but I just tried it out locally, and it failed with an OOM error. I'd look into it more, but brain power running out this late at night. I'm attaching a file with the scrub errors. Will look into it tomorrow.
          Hide
          Jonathan Ellis added a comment -

          Broke out scrub into CASSANDRA-5429 so we can review this separately.

          Show
          Jonathan Ellis added a comment - Broke out scrub into CASSANDRA-5429 so we can review this separately.
          Hide
          Jonathan Ellis added a comment -

          Pushed updated code to http://github.com/jbellis/cassandra/tree/4180-3. Besides the rebase, this cleans things up by making ColumnIndex the sole entity responsible for writing the row header (thus removing the flag I'd added to it in the earlier version).

          Show
          Jonathan Ellis added a comment - Pushed updated code to http://github.com/jbellis/cassandra/tree/4180-3 . Besides the rebase, this cleans things up by making ColumnIndex the sole entity responsible for writing the row header (thus removing the flag I'd added to it in the earlier version).
          Hide
          Jason Brown added a comment -

          On the whole, lgtm. Tests, especially scrub, are working correctly now. I do appreciate the row header creation being centralized now; that was tricky for me when I first started working on this.

          However, when I create a simple table under 1.2, then go to start this trunk, it fails on launch with:

           INFO [main] 2013-04-17 06:49:47,456 CacheService.java (line 165) Scheduling row cache save to each 0 seconds (going to save all keys).
           INFO [SSTableBatchOpen:2] 2013-04-17 06:49:47,710 SSTableReader.java (line 168) Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-1 (261 bytes)
           INFO [SSTableBatchOpen:1] 2013-04-17 06:49:47,710 SSTableReader.java (line 168) Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-2 (165 bytes)
           INFO [SSTableBatchOpen:1] 2013-04-17 06:49:48,121 SSTableReader.java (line 168) Opening /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ib-3 (691 bytes)
           INFO [SSTableBatchOpen:2] 2013-04-17 06:49:48,121 SSTableReader.java (line 168) Opening /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ib-1 (4540 bytes)
           INFO [SSTableBatchOpen:3] 2013-04-17 06:49:48,122 SSTableReader.java (line 168) Opening /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ib-2 (699 bytes)
           INFO [SSTableBatchOpen:1] 2013-04-17 06:49:48,144 SSTableReader.java (line 168) Opening /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-2 (209 bytes)
           INFO [SSTableBatchOpen:2] 2013-04-17 06:49:48,145 SSTableReader.java (line 168) Opening /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-3 (194 bytes)
           INFO [SSTableBatchOpen:3] 2013-04-17 06:49:48,145 SSTableReader.java (line 168) Opening /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-1 (3768 bytes)
          INFO [SSTableBatchOpen:1] 2013-04-17 06:49:48,181 SSTableReader.java (line 168) Opening /var/lib/cassandra/data/system/local/system-local-ib-5 (441 bytes)
          ERROR [main] 2013-04-17 06:49:48,652 CassandraDaemon.java (line 454) Exception encountered during startup
          java.lang.OutOfMemoryError: Java heap space
          	at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:339)
          	at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)
          	at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
          	at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:124)
          	at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:82)
          	at org.apache.cassandra.db.Column$1.computeNext(Column.java:73)
          	at org.apache.cassandra.db.Column$1.computeNext(Column.java:62)
          	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
          	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
          	at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:92)
          	at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:36)
          	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
          	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
          	at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:90)
          	at org.apache.cassandra.db.columniterator.LazyColumnIterator.computeNext(LazyColumnIterator.java:82)
          	at org.apache.cassandra.db.columniterator.LazyColumnIterator.computeNext(LazyColumnIterator.java:59)
          	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
          	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
          	at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:136)
          	at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:119)
          	at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:199)
          	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
          	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
          	at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:131)
          	at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:101)
          	at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:75)
          	at org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:105)
          	at org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:78)
          	at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114)
          	at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
          	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
          	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
          

          I'll try to take a look today.

          Show
          Jason Brown added a comment - On the whole, lgtm. Tests, especially scrub, are working correctly now. I do appreciate the row header creation being centralized now; that was tricky for me when I first started working on this. However, when I create a simple table under 1.2, then go to start this trunk, it fails on launch with: INFO [main] 2013-04-17 06:49:47,456 CacheService.java (line 165) Scheduling row cache save to each 0 seconds (going to save all keys). INFO [SSTableBatchOpen:2] 2013-04-17 06:49:47,710 SSTableReader.java (line 168) Opening / var /lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-1 (261 bytes) INFO [SSTableBatchOpen:1] 2013-04-17 06:49:47,710 SSTableReader.java (line 168) Opening / var /lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-2 (165 bytes) INFO [SSTableBatchOpen:1] 2013-04-17 06:49:48,121 SSTableReader.java (line 168) Opening / var /lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ib-3 (691 bytes) INFO [SSTableBatchOpen:2] 2013-04-17 06:49:48,121 SSTableReader.java (line 168) Opening / var /lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ib-1 (4540 bytes) INFO [SSTableBatchOpen:3] 2013-04-17 06:49:48,122 SSTableReader.java (line 168) Opening / var /lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ib-2 (699 bytes) INFO [SSTableBatchOpen:1] 2013-04-17 06:49:48,144 SSTableReader.java (line 168) Opening / var /lib/cassandra/data/system/schema_columns/system-schema_columns-ib-2 (209 bytes) INFO [SSTableBatchOpen:2] 2013-04-17 06:49:48,145 SSTableReader.java (line 168) Opening / var /lib/cassandra/data/system/schema_columns/system-schema_columns-ib-3 (194 bytes) INFO [SSTableBatchOpen:3] 2013-04-17 06:49:48,145 SSTableReader.java (line 168) Opening / var /lib/cassandra/data/system/schema_columns/system-schema_columns-ib-1 (3768 bytes) INFO [SSTableBatchOpen:1] 2013-04-17 06:49:48,181 SSTableReader.java (line 168) Opening / var /lib/cassandra/data/system/local/system-local-ib-5 (441 bytes) ERROR [main] 2013-04-17 06:49:48,652 CassandraDaemon.java (line 454) Exception encountered during startup java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:339) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392) at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355) at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:124) at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:82) at org.apache.cassandra.db.Column$1.computeNext(Column.java:73) at org.apache.cassandra.db.Column$1.computeNext(Column.java:62) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:92) at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:36) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:90) at org.apache.cassandra.db.columniterator.LazyColumnIterator.computeNext(LazyColumnIterator.java:82) at org.apache.cassandra.db.columniterator.LazyColumnIterator.computeNext(LazyColumnIterator.java:59) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:136) at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:119) at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:199) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:131) at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:101) at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:75) at org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:105) at org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:78) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) I'll try to take a look today.
          Hide
          Jason Brown added a comment -

          I can confirm that creating same table under trunk, and relaunching, works correctly, as well as basic select querying. So I think it's something in the upgrade path from 1.2 to trunk.

          Show
          Jason Brown added a comment - I can confirm that creating same table under trunk, and relaunching, works correctly, as well as basic select querying. So I think it's something in the upgrade path from 1.2 to trunk.
          Hide
          Jason Brown added a comment -

          Just more info: tried my 1.2 to trunk upgrade again, and it's not related to my table at all. I simply started 1.2, let it create the system tables, killed it, and launched trunk - same exception. Will check it out.

          Show
          Jason Brown added a comment - Just more info: tried my 1.2 to trunk upgrade again, and it's not related to my table at all. I simply started 1.2, let it create the system tables, killed it, and launched trunk - same exception. Will check it out.
          Hide
          Jonathan Ellis added a comment -

          Rebased on top of CASSANDRA-5511 with a fix or two: http://github.com/jbellis/cassandra/tree/4180-4

          Now passes the "start 1.2, rebuild and start 4180-4" test.

          Show
          Jonathan Ellis added a comment - Rebased on top of CASSANDRA-5511 with a fix or two: http://github.com/jbellis/cassandra/tree/4180-4 Now passes the "start 1.2, rebuild and start 4180-4" test.
          Hide
          Jonathan Ellis added a comment -
          Show
          Jonathan Ellis added a comment - Rebased again, to https://github.com/jbellis/cassandra/commits/4180-5 . Marcus Eriksson , can you review?
          Hide
          Jonathan Ellis added a comment -
          Show
          Jonathan Ellis added a comment - Minor rebase to https://github.com/jbellis/cassandra/commits/4180-6
          Hide
          Marcus Eriksson added a comment -

          in general, lgtm, few issues;

          sstable.getPosition(..) can return null in a couple of cases:
          During repair:

          ERROR [ValidationExecutor:2] 2013-05-07 10:43:24,875 CassandraDaemon.java (line 179) Exception in thread Thread[ValidationExecutor:2,1,main]
          java.lang.NullPointerException
                  at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:109)
                  at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1038)
                  at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:214)
                  at org.apache.cassandra.db.compaction.CompactionManager$ValidationCompactionIterable.<init>(CompactionManager.java:744)
                  at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:648)
                  at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:64)
                  at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:391)
                  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
                  at java.lang.Thread.run(Thread.java:722)
          

          and scrub is not supposed to work until CASSANDRA-5429 right?:

           INFO 09:26:43,968 Retrying from row index; data is 261 bytes starting at 295
           WARN 09:26:43,969 Retry failed too. Skipping to next row (retry's stacktrace follows)
          java.lang.AssertionError: Interval min > max
                  at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:250)
                  at org.apache.cassandra.utils.IntervalTree.<init>(IntervalTree.java:72)
                  at org.apache.cassandra.utils.IntervalTree.build(IntervalTree.java:81)
                  at org.apache.cassandra.db.DeletionInfo.add(DeletionInfo.java:179)
                  at org.apache.cassandra.db.AbstractThreadUnsafeSortedColumns.delete(AbstractThreadUnsafeSortedColumns.java:44)
                  at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:142)
                  at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:177)
                  at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:184)
                  at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114)
                  at org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:98)
                  at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:220)
                  at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:226)
                  at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:204)
                  at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:430)
                  at org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:419)
                  at org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:64)
                  at org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:234)
                  at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:220)
                  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
                  at java.lang.Thread.run(Thread.java:722)
           WARN 09:26:43,972 Non-fatal error reading row (stacktrace follows)
          java.io.IOError: java.io.IOException: Impossible row size 9223372034707292160
                  at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:170)
                  at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:430)
                  at org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:419)
                  at org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:64)
                  at org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:234)
                  at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:220)
                  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
                  at java.lang.Thread.run(Thread.java:722)
          Caused by: java.io.IOException: Impossible row size 9223372034707292160
                  ... 11 more
          

          nits;

          • version argument unused in IndexedSliceReader#setToRowStart
          • LazilyCompactedRow comment obsolete
          • compactionType unused param in SSTableWriter#createWriter
          Show
          Marcus Eriksson added a comment - in general, lgtm, few issues; sstable.getPosition(..) can return null in a couple of cases: During repair: ERROR [ValidationExecutor:2] 2013-05-07 10:43:24,875 CassandraDaemon.java (line 179) Exception in thread Thread [ValidationExecutor:2,1,main] java.lang.NullPointerException at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:109) at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1038) at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:214) at org.apache.cassandra.db.compaction.CompactionManager$ValidationCompactionIterable.<init>(CompactionManager.java:744) at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:648) at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:64) at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:391) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang. Thread .run( Thread .java:722) and scrub is not supposed to work until CASSANDRA-5429 right?: INFO 09:26:43,968 Retrying from row index; data is 261 bytes starting at 295 WARN 09:26:43,969 Retry failed too. Skipping to next row (retry's stacktrace follows) java.lang.AssertionError: Interval min > max at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:250) at org.apache.cassandra.utils.IntervalTree.<init>(IntervalTree.java:72) at org.apache.cassandra.utils.IntervalTree.build(IntervalTree.java:81) at org.apache.cassandra.db.DeletionInfo.add(DeletionInfo.java:179) at org.apache.cassandra.db.AbstractThreadUnsafeSortedColumns.delete(AbstractThreadUnsafeSortedColumns.java:44) at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:142) at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:177) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:184) at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114) at org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:98) at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:220) at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:226) at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:204) at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:430) at org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:419) at org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:64) at org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:234) at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:220) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang. Thread .run( Thread .java:722) WARN 09:26:43,972 Non-fatal error reading row (stacktrace follows) java.io.IOError: java.io.IOException: Impossible row size 9223372034707292160 at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:170) at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:430) at org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:419) at org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:64) at org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:234) at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:220) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang. Thread .run( Thread .java:722) Caused by: java.io.IOException: Impossible row size 9223372034707292160 ... 11 more nits; version argument unused in IndexedSliceReader#setToRowStart LazilyCompactedRow comment obsolete compactionType unused param in SSTableWriter#createWriter
          Hide
          Jonathan Ellis added a comment -

          Pushed fixes to -6 branch. Yes, scrub is expected to be totally broken at this point.

          Show
          Jonathan Ellis added a comment - Pushed fixes to -6 branch. Yes, scrub is expected to be totally broken at this point.
          Hide
          Marcus Eriksson added a comment -

          ok, this lgtm!

          Show
          Marcus Eriksson added a comment - ok, this lgtm!
          Hide
          Jonathan Ellis added a comment -

          committed!

          Show
          Jonathan Ellis added a comment - committed!

            People

            • Assignee:
              Jonathan Ellis
              Reporter:
              Sylvain Lebresne
              Reviewer:
              Marcus Eriksson
            • Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development