[KUDU-3023] Enforce consistency between maximum RPC size and maximum size of transaction

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.12.0
• Component/s: master, tserver
• Labels: Availability consistency scalability
• Target Version/s: 1.12.0
• Code Review: http://gerrit.cloudera.org:8080/15030
• Assignee: Alexey Serbin
• Reporter: Alexey Serbin
• Created: 17/Dec/19 04:35
• Updated: 15/Jan/20 00:41
• Resolved: 15/Jan/20 00:41

Description

It seems that with the absence of constraints for the --rpc_max_message_size and the --tablet_transaction_memory flags, there might a transaction that can get into the WALs of tablet replicas, but no replica can apply the transaction.

It would be nice to clarify on this and if necessary, establish a group validator to prevent such situations from happening, if possible.

Few other points to clarify:

• What if the data in RPC message that client sends to leader replica is compressed? Can we guarantee that corresponding Raft update sent to followers after persisting the data in the leader's WAL is compressed as well?
• How much cruft is added by converting the incoming data into a WAL entry and then into a Raft update RPC message? Is it possible for a leader replica to accept an update on the tablet since it's under the max RPC size limit, but be unable push corresponding Raft update message to follower replicas because of some extra cruft added when converting the accepted WriteRequestPB into ConsensusRequestPB containing ReplicateMsg with embedded original WriteRequestPB?