[CASSANDRA-16442] Improve handling of failed prepared statement loading

• Type: Improvement
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 3.11.x
• Component/s: Cluster/Schema
• Change Category: Operability
• Complexity: Low Hanging Fruit
• Platform: All
• Assignee: Unassigned
• Reporter: Thomas Steinmaurer
• Created: 12/Feb/21 07:03
• Updated: 12/Feb/21 16:41

Description

In an internal DEV cluster, when going from 3.0 to 3.11 we have seen the following WARN logs constantly upon Cassandra startup.

...
WARN [main] 2021-02-05 09:25:06,892 QueryProcessor.java:160 - prepared statement recreation error: SELECT n,v FROM "Ts2Volatile60Min" WHERE k=? LIMIT ?;
WARN [main] 2021-02-05 09:25:06,895 QueryProcessor.java:160 - prepared statement recreation error: INSERT INTO "Ts2Final01Min" (k,n,v) VALUES (?,?,?) USING TIMESTAMP ?;
...

I guess 3.11 tries to pre-load prepared statements for tables which don't exist anymore. On how we got into this situation was our fault I think (Cas 3.0 => Upgrade 3.11 => Downgrade 3.0 => with 3.0 some tables got dropped => Upgrade 3.11.10).

Still, perhaps there is room for improvement when it comes to loading persisted prepared statements, which might fail.

I thought about:

• An additional nodetool option to wipe the persisted prepared statement cache
• Perhaps even make the startup code smarter in a way, when loading of a prepared statement fails, due to a table not being available anymore, then auto-wipe such entries from the prepared_statements system table

To get rid of the WARN log, I currently need to work directly on the "prepared_statements" system table, but I don't know if it is safe to run e.g. a TRUNCATE statement, thus currently, it seems we need to take each node offline, execute a Linux rm command on SSTables for the system table.