Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Problem
Currently, there are two places where Command instances are being serialized:
- ActionRequest - here the command property is marked is Marshallable, meaning that it will be serialized using a User Object Serialization approach
- Listener - here command is explicitly serialized using JDKMarshaller, for further handling by RAFT. This is the data that will be written to the Log and deserialized on followers / learners
What are the problems?
For ActionRequest message, command is expected to be its largest part. And although writing serialized UOS byte array into a netty socket is faster, then optimized marshalling, it feels like overall throughput will be smaller. And the reason is that there's an extra step of converting command into a byte array, that happens in caller thread.
For serialization in listeners, using JDKMarshaller is both slow and inefficient in terms of space. Obvious example - network serialization of SnapshotMeta object, for instance, can be condensed to 8 bytes (assuming we optimize "writeShort" and change its message type). JDKMarshaller produces 232 bytes. Of course, here most of fields are nulls and real payload will be bigger, but JDKMarshaller will always lead to more data simply because it has to store schema meta-information.
Solution
Making Command an implementation of NetworkMessage will solve both of these problems. ActionRequest will not have its "prepareMarshal" phase, listeners will have fast and space-efficient serialization algorithm.
Of course, there must be drawbacks. I'll try to explain what I see at the moment.
- Currently, there's no explicit support for List properties, only Collection. It is easy to fix
- CMG commands use classes like ClusterNode and IgniteProductVersion. We should introduce message alternatives
- I saw some enums being used, they are not natively supported at the moment. There are two options:
- implement native support. I consider this a dangerous path
- store explicit ordinal where it's necessary
- ByteBuffer support would be really nice to have natively. Should be fast to implement also
One important note: there should be no Marshallable properties in commands, because we can't persist them. Information about classes' ids is stored in sessions and can change between sessions. The way to achieve it is to pass a "null" UOS context into serializator.
Now about serializator: we can have thread-local buffers to write data to. When write is complete, data is copied as a byte[]. Reading will be done directly from the byte[].
Possible optimization for ByteBuffers - we can implement them as slices of the byte[] payload instead of copying sub-arrays. Will save some time and memory.
Plan
Given the volume of changes, I suggest splitting the issue into several parts. There are multiple sets of commands in Ignite:
- Table commands (5 commands at the moment of writing this text)
- CMG commands (6 commands)
- Metastorage (19)
This list goes in order of complexity. Table commands are very simple. CMG commands require additional messages for ClusterNodes and such.
Metastorage commands have complicated structures for conditional updates, and there are many of them.
When all commands are messages, we can safely inherit Command from NetworkMessage and remove Marshallable from the ActionRequest's field. In total, this looks like 4 separate issues, 4th one being the current one. List & ByteBuffer support is already completed in IGNITE-17874.
Of course, implementation of the OptimizedMarshaller should be done here as well. I'd also provide some ideas to whoever's going to implement the improvement, there's at least one possible optimization in mind that's hard to put in words.
There are also tests, they can be migrated in current issue as well.
Attachments
Issue Links
- requires
-
-
- Resolved
-
-
-
- Resolved
-
-
-
- Resolved
-
-
-
- Resolved
-
-
-
- Resolved
-
- links to
