[IGNITE-21283] Improve client data streamer performance

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-beta1
• Fix Version/s: 3.0.0-beta2
• Component/s: thin client
• Labels: ignite-3
• Epic Link: AI3 Performance
• Assignee: Pavel Tupitsyn
• Reporter: Pavel Tupitsyn
• Created: 17/Jan/24 10:27
• Updated: 08/Feb/24 06:21

Description

• Embedded data streamer uses partition-based org.apache.ignite.internal.table.InternalTable#upsertAll(java.util.Collection<org.apache.ignite.internal.schema.BinaryRowEx>, int) method to upsert data, but client streamer uses upsertAll through public API
• Embedded streamer performs batching per partition, client streamer batches per node (connection)

Compare embedded and client streamer performance in different cases:

• 1 node, 4 nodes
• client connects to single node, client connects to all nodes

Check if using per-partition approach with internal upsertAll API improves client streamer performance.

UPDATE

IGNITE-21402 changed the logic to use per-partition batches:

• Compare client and embedded streamer performance
• Check if per-node is better for client
• Rename perNodeParallelOperations if necessary

Issue Links

• is related to: IGNITE-21402 DataStreamer data removal: refactor client and embedded streamers to use the same internal API