expanding Streaming is fine, but it's not as straightforward as you might think because it is completely based around FileChannel.transferTo, which on Linux at least is a thin wrapper around the sendfile system call

Integrated in Cassandra #278 (See http://hudson.zones.apache.org/hudson/job/Cassandra/278/)

Bumping priority on this one, because goffinet experienced a repair taking > 24 hours for ~100GB of data.

Also note, this was on 20 CFs

We see about 16MB/s for compaction on a 4 disk RAID0 configuration. Moving this directly to network, so we can be network bound would be much better. But the thing is, compaction is running slower than the max throughput on network, which is roughly 80MB/s on our systems.

What you want to be careful about here is not screwing up the FileChannel.transferTo optimization which is very valuable.

So IMO what you want to do is:

1. use SSTR.getPosition to find start and end ranges to transfer, then use the existing streaming API – which already support streaming only parts of files via transferTo – to send that over as the data file in question.

2. from the data file, compute index + BF files on the destination node, instead of wasting IO streaming those from the source.

• streaming the index from source is possible, but since you need to scan the data file anyway to build BF (since there is no way to extract a subset of a BF) I think it's going to be simpler to just rebuild both. And anyway goffinet has wanted a "rebuild index from data file" for a while now

> 1. use SSTR.getPosition to find start and end ranges to transfer...
I don't think this works: you still need to perform the compaction first, which means you still need to dump to disk.
EDIT: Unless you are suggesting streaming each SSTable individually?

> 2. from the data file, compute index + BF files on the destination node
This seems like the best approach.

> Unless you are suggesting streaming each SSTable individually?

streaming each range individually (so, one source sstable since you just did a major compact, and each range becomes an sstable on the target)

> ...since you just did a major compact...
You didn't necessarily just do a major compaction: currently, when someone triggers a repair manually, it performs a readonly major compaction, which doesn't write to disk.

What if we just stream from individual SSTables directly? That would of course send some stuff twice (or more) if done without compaction, but that seems like small overhead. If we streamed directly from SSTables, that would allow us to bootstrap from a node that is short of disk space.

> What if we just stream from individual SSTables directly?
This idea is growing on me... it causes minimal load on the machine that you are trying to move data off of, in terms of CPU and disk (compaction is done on the destination side). Network bandwidth should be relatively plentiful.

one side has to compact eventually, so it should be the sending side that compacts, since that gets you reduced bandwidth; there's no benefit by having receiver compact. it would be quite easy to compact to a socket stream instead of a disk one. (you lose transferTo, but receiver can still do transferFrom. and we could special case only-a-single-source-sstable with transferTo if that is worth it...)

Note that if we're streaming w/o first writing temporary file locally there is less reason to keep all outbound streaming single-threaded, but Streaming contexts currently assume there can only be one outstanding stream per source node. So keep that in mind if we are tempted to multithread streaming.

I think we should delay making too many changes to streaming until we've finalized the SSTable versioning/interface changes proposed on #674. All of the possible approaches to optimizing this depend on the file format. For instance, sending only portions of the file depends on a splittable format, and performing the compaction on the sending side and then writing a new SSTable on the receiving side depends on the format of the serialized CompactedRows that would be sent across.

We can probably resume progress here now that 389 is in, although I hope to get 777 in first.

Tons of people get bitten by this, and everyone else is lagged out by slow transfers.

Given how much cpu the compact part of anticompact chew up, Stu is right that we shouldn't do that on the source node. But we shouldn't do it on the target node, either, or more precisely, we shouldn't do it as part of the stream operation and let the operator do it manually if desired.

We should use the row index to figure out what parts of the source data files to stream over, and send those chunks w/o deserializing anything. On the target side, we should read the keys but not deserialize anything else, and build the index + bloom filter from that.

Here is a patchset for trunk implementing the "stream matching ranges from existing sstables, and then rebuild indexes" strategy. It applies on top of 1117 (for no good reason, in retrospect). It passes existing tests, but I haven't tried it out on a cluster: I'll give that a shot later today.

0005 Needs more work.

Rebased for trunk: should be ready for review now.

Rebased for trunk.

// FIXME: extremely rough estimate of keys in the sstable

I think we do need a better key count estimate. What about reading the first 1% (by size) of the rows, and estimating from that?

Alternatively streamin could actually count the rows as they arrive.

patch to fix row size post- CASSANDRA-16.

probably worth exploring why StreamingTest doesn't catch this.

1. Squashed your patch into 0002
2. Estimated keys in 0002 based on the first 100 keys or 100 megabytes of data, whichever comes first
3. Fixed the test in 0005 to catch the long/int discrepancy: StreamingTest only tested recovering a single row
4. Removed getNearestPosition in 0005

patches 01-03 applied, but I keep getting timeouts trying to commit the rest. (probably the super flaky wireless here.) will try again in the morning.

done. (squashed 01-03 on purpose; 04 and 05 apparently got squashed somehow by git-svn getting confused during repeated borked dcommits.)

Integrated in Cassandra #469 (See http://hudson.zones.apache.org/hudson/job/Cassandra/469/)
Stream sstables without anticompaction
patch by Stu Hood; reviewed by jbellis for CASSANDRA-579
Extract index/filter writing into IndexWriter; add recovery for non-essential sstable components; only send the datafile when streaming
patch by Stu Hood; reviewed by jbellis for CASSANDRA-579