[ACCUMULO-1754] support scale-up behavior in BatchWriter

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: client
• Assignee: Unassigned
• Reporter: Adam Fuchs
• Created: 07/Oct/13 17:27
• Updated: 07/Oct/13 20:28

Description

The o.a.a.client.impl.TabletServerBatchWriter currently limits write threads to one per server. Some people want to use Accumulo on beefy servers where sending one batch at a time is a bottleneck. We can improve the performance on small clusters of beefy nodes by splitting up write batches by tablet and server.

This should be done in such a way that we can still batch across tablets when the cluster is large. Basically, mutations should be binned by server, then further binned by tablet up to the limit of (some factor of) threads in the BatchWriter's thread pool.

Issue Links

• relates to: ACCUMULO-1083 add concurrency to HDFS write-ahead log