Major Description
Currently, the Kudu C++ client buffers incoming operations regardless of their destination tablet server. Accordingly, it's possible to set limit on the total buffer space, not per tablet server. This approach works but there is room for improvement: there are real-world scenarios where per-TS buffering would be more robust. Besides, tablet servers impose limit on the RPC operations size.
Grouping write operations on per-tablet-server basis would be beneficial for 'one-out-of-many lagging tablet server' scenario. There, all tablet servers for a table perform well except for one which runs slow due to excessive IO, network issues, failing disk, etc. The problem is that the lagging server hinders the overall performance. This is due to the current approach to the buffer turnaround: a buffer is considered 'flushed' and its space is reclaimed at once when all operations in the buffer are completed. So, if 1000 operations have already been sent but there is 1 operation still in progress, the whole buffer space is 'locked' and cannot be used.
Accordingly, introducing per-tablet-server buffer limit would help to address scenarios with concurrent writes into tables with extremely diverse partition factors (like 2 and 100). E.g., consider a case when incoming write operations for tables with diverse partition factors are intermixed in the context of one session. The problem is that setting the total buffer space limit high is beneficial for the writes into the table with many partitions (assuming those writes are evenly distributed across participating tablets), but it may be over the server-side's limit for max transaction size if those writes are targeted for the table with a few partitions.