[KUDU-3020] Add metric gauges for the size of incoming RPC request payload and number of RPC rejections due to payload size

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: master, metrics, tserver
• Labels: guidelines observability scalability troubleshooting
• Assignee: Unassigned
• Reporter: Alexey Serbin
• Created: 13/Dec/19 03:33
• Updated: 03/Jun/20 15:58

Description

Kudu servers have a limit on the size of RPC's payload they accept: --rpc_max_message_size.

It would be nice to introduce corresponding metrics to gauge how relevant the current setting for the maximum RPC size is with regard to the incoming requests. That can help with pro-active tuning of a Kudu cluster to sustain a planned increase in workload. This is especially useful for tuning parameters of Kudu masters to accommodate higher number of tables/tablets in a cluster (e.g., adding new tables or creating new partitions for already existing tables).