[CASSANDRA-6855] Native protocol V3

• Type: New Feature
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.1 beta2
• Component/s: Legacy/CQL
• Labels: qa-resolved
• Assignee: Sylvain Lebresne
• Reporter: Sylvain Lebresne

Authors

Sylvain Lebresne

Reviewers

Tom Hobbs

• Tester: Michael Shuler
• Created: 14/Mar/14 13:07
• Updated: 16/Apr/19 09:31
• Resolved: 01/May/14 09:45

Description

I think we need a V3 of the protocol for 2.1. The things that this could/should includes are:

1. Adding an optional Serial CL for protocol batches (like we have for QUERY and EXECUTE). It was an oversight of V2 of not adding it, and now that we can batch conditional updates, it's definitively missing.
2. Proper type codes for UDT. This is not strictly needed to be able to support UDT since currently a UDT will be sent as a "custom type" with his fully class name + arguments. But parsing that is no fun nor convenient for clients. It's also not particular space efficient (though that's probably not a huge concern since with prepared statement you can avoid sending the ResultSet metadata every time).
3. Serialization format for collections. Currently the serialization format only allow for 65K elements, each of 65K bytes size at most. While collections are not meant to store large amount of data, having the limitation in the protocol serialization format is the wrong way to deal with that. Concretely, the current workaround for CASSANDRA-5428 is ugly. I'll note that the current serialization format is also an obstacle to supporting null inside collections (whether or not we want to support null there is a good question, but here again I'm not sure being limited by the serialization format is a good idea).
4. CASSANDRA-6178: I continue to believe that in many case it makes somewhat more sense to have the default timestamp provided by the client (this is a necessary condition for true idempotent retries in particular). I'm absolutely fine making that optional and leaving server-side generated timestamps by default, but since client can already provide timestamp in query string anyway, I don't see a big deal in making it easier for client driver to control that without messing with the query string.
5. Optional names for values in QUERY messages: it has been brought to my attention that while V2 allows to send a query string with values for a one-roundtrip bind-and-execute, a driver can't really support named bind marker with that feature properly without parsing the query. The proposition is thus to make it (optionally) possible to ship the name of the marker each value is supposed to be bound to.

I think that 1) and 2) are enough reason to make a V3 (even if there is disagreement on the rest that is).

3) is a little bit more involved tbh but I do think having the current limitations bolted in the protocol serialization format is wrong in the long run, and it turns out that due to UDT we will start storing serialized collections internally so if we want to lift said limitation in the serialization format, we should do it now and everywhere, as doing it afterwards will be a lot more painful.

4) and 5) are probably somewhat more minor, but at the same time, both are completely optional (a driver won't have to support those if he doesn't want). They are really just about making things more flexible for client drivers and they are not particularly hard to support so I don't see too many reasons not to include them.

Last but not least, I know that some may find it wrong to do a new protocol version with each major of C*, so let me state my view here: I fully agree that we shouldn't make an habit of that in the long run and that's definitively not my objective. However, it would be silly to expect that we could get everything right and forget nothing in the very first version. It shouldn't be surprising that we'll have to burn a few versions (and there might be a few more yet) before getting something more stable and complete and I think that delaying the addition of stuffs that are useful to create some fake notion of stability would be even more silly. On the bright side, the additions of this V3 are comparatively much more simple to implement for a client that those of V2 (in fact, for clients that want to support UDT, it will probably require less effort to add the changes for this new version than to try to support UDT without it), so I do think we make good progress on getting the protocol stabilized

Issue Links

• breaks: CASSANDRA-7267 Embedded sets in user defined data-types are not updating
• is duplicated by: CASSANDRA-6428 Use 4 bytes to encode collection size in next native protocol version