This test goes outside the norm for Cassandra, creating ~2000 column families, and writing large text blobs to them.
The process goes like this:
Bring up a 6 node m2.2xlarge cluster on EC2. This instance type has enough memory (34.2GB) so that Cassandra will allocate a full 8GB heap without tuning cassandra-env.sh. However, this instance type only has a single drive, so data and commitlog are comingled. (This test has also been run m1.xlarge instances which have four drives (but lower memory) and has exhibited similar results when assigning one to commitlog and 3 to datafile_directories.)
Use the 'memtable_allocator: HeapAllocator' setting from
Create 2000 CFs:
This process of creating tables takes a long time, about 5 hours, but for anyone wanting to create that many tables, presumably they only need to do this once, so this may be acceptable.
The test dataset consists of writing 100K, 1M, and 10M documents to these tables:
With 5 threads doing these inserts across the cluster, indefinitely, randomly choosing a table number 1-2000, the cluster eventually topples over with 'OutOfMemoryError: Java heap space'.
A heap dump analysis indicates that it's mostly memtables:
Best current theory is that this is commitlog bound and that the memtables cannot flush fast enough due to locking issues. But I'll let Jonathan Ellis comment more on that.