Normal Description
Upgrading a cluster from 1.1.12 -> 1.2.13 -> 2.0 HEAD fails due to upgradesstables not upgrading the index files.
To reproduce:
# Make sure ccm has all the versions we need: ccm create -v git:cassandra-2.0 test ccm remove ccm create -v git:cassandra-1.2.13 test ccm remove # Create a 1.1.12 cluster: ccm create -v git:cassandra-1.1.12 test # Set cluster partitioner: perl -p -i -e 's/partitioner: null/partitioner: RandomPartitioner/gi' ~/.ccm/test/cluster.conf ccm populate -n 1 ccm start ccm node1 stress -- --operation=INSERT --family-type=Standard --num-keys=10000 --create-index=KEYS --compression=SnappyCompressor --compaction-strategy=LeveledCompactionStrategy ccm flush ccm node1 drain ccm status # Wait until node1 shows DOWN. # Set cluster version: perl -p -i -e 's/git_cassandra-1.1.12/git_cassandra-1.2.13/gi' ~/.ccm/test/cluster.conf # Upgrade node1: ccm node1 updateconf ccm node1 start # Upgrade sstables: ~/.ccm/test/node1/bin/nodetool -p 7100 upgradesstables ls ~/.ccm/test/node1/data/Keyspace1/Standard1/ # Note the versions on files. Data has been upgraded to version *ic* but indexes are left on version *hf*. # Upgrade to 2.0: ccm flush ccm node1 drain ccm status # Wait until node1 shows DOWN. # Set cluster version: perl -p -i -e 's/git_cassandra-1.2.13/git_cassandra-2.0/gi' ~/.ccm/test/cluster.conf ccm node1 updateconf ccm node1 start
On this last upgrade attempt, cassandra 2.0 complains that the version for the indexes is incorrect:
java.lang.RuntimeException: Can't open incompatible SSTable! Current version jb, found file: /home/ryan/.ccm/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1.Idx1-hf-1
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:411)
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:393)
at org.apache.cassandra.db.index.AbstractSimplePerColumnSecondaryIndex.init(AbstractSimplePerColumnSecondaryIndex.java:52)
at org.apache.cassandra.db.index.SecondaryIndexManager.addIndexedColumn(SecondaryIndexManager.java:274)
at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:279)
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:416)
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:388)
at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309)
at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266)
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110)
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:273)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:462)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:549)
The same test can be done starting from 1.2 upgrading to 2.0. The index files do not upgrade in this scenario either, however, there is not the same error, possibly because 2.0 is tolerant of version 1.2 indexes?