[CASSANDRA-16257] When upgrading from 2.1.0 to 2.2.0, UnknownColumnFamilyException encountered on upgraded nodes

• Type: Bug
• Status: Triage Needed
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 2.2.x
• Platform: All
• Assignee: Unassigned
• Reporter: Zhuqi Jin
• Created: 10/Nov/20 01:04
• Updated: 15/Jan/23 09:20

Description

Both using 2.1.0 node as seed and using 2.2.0 node as seed will fail with this similar error:

org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=49a44701-d6bc-11ea-848f-4766f428c026 (when using 2.1.0 node as seed)
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=727d9450-d6c7-11ea-9370-4766f428c026 (when using 2.2.0 node as seed)at org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:165) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(IncomingTcpConnection.java:147) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) ~[apache-cassandra-2.1.0.jar:2.1.0]

This seems to originate from the difference in the settings in the stress tests, and is not a data format bug. The field `cfId` is a UUID string generated from another two fields: `ksName` and `cfName`. These two names are not the interior settings in the cluster, instead they come from user specification.

In 2.1.0, the stress test configures these two names to be “Keyspace1” and “Standard1”. See https://github.com/apache/cassandra/blob/cassandra-2.1.0/tools/stress/src/org/apache/cassandra/stress/settings/SettingsSchema.java#L84 . Here is my log:

WARN  [MigrationStage:1] 2020-08-05 03:17:20,514 Schema.java:330 - Schema.load: Adding org.apache.cassandra.config.CFMetaData@6299796f…..., key = (Keyspace1,Standard1), val = 49a44701-d6bc-11ea-848f-4766f428c026

In 2.2.0, the stress test configures these two names to be “keyspace1” and “standard1” (note that the names are case-sensitive). See https://github.com/apache/cassandra/blob/cassandra-2.2.0/tools/stress/src/org/apache/cassandra/stress/settings/SettingsSchema.java#L225  . Here is my log:

WARN  [MigrationStage:1] 2020-08-05 02:57:49,680 Schema.java:350 - Schema.load: Adding org.apache.cassandra.config.CFMetaData@719822c2……..., key = (keyspace1,standard1), val = 727d9450-d6c7-11ea-9370-4766f428c026

When a 2.2.0 message reaches a 2.1.0 node, or vice versa, the receiver will look up the `cfId` field in a map. Since the key is not present in its own map, the look up will fail with the above message. Unluckily this problem is quite hard to fix, since many other places also depend on the name. So I can not overcome this problem and find actual data format bugs between these two versions currently.