I suggest to configure populate_io_cache_on_flush option for each column family. It should be configurable from cassandra-cli and should be stored in System keyspace.
That could be useful if you have a few column families inside single keyspace and you need to fit in memory only one of them.
Patch has been attached. I've been testing it on pseudo-cluster using ccm. So I don't have fully confidence about lack of bugs. Please carefully review that code.