[HBASE-6295] Possible performance improvement in client batch operations: presplit and send in background

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.95.2
• Fix Version/s: 0.98.0, 0.95.2
• Component/s: Client, Performance
• Hadoop Flags: Reviewed

Release Note

The puts are now streamed, i.e. sent asynchronously to the region servers if autoflush it set to false. If a region server is slow or does not respond, its puts are kept into the write buffer while the others are sent to these respective region server, until the write buffer is full. This feature is keeps the semantic of the interface already existing in 0.94 when using autoflush.

• Assignee: Nicolas Liochon
• Reporter: Nicolas Liochon
• Created: 30/Jun/12 11:45
• Updated: 05/Aug/14 20:11
• Resolved: 24/Jun/13 19:04

Description

today batch algo is:

for Operation o: List<Op>{
  add o to todolist
  if todolist > maxsize or o last in list
    split todolist per location
    send split lists to region servers
    clear todolist
    wait
}

We could:

• create immediately the final object instead of an intermediate array
• split per location immediately
• instead of sending when the list as a whole is full, send it when there is enough data for a single location

It would be:

for Operation o: List<Op>{
  get location
  add o to todo location.todolist
  if (location.todolist > maxLocationSize)
    send location.todolist to region server 
    clear location.todolist
    // don't wait, continue the loop
}
send remaining
wait

It's not trivial to write if you add error management: retried list must be shared with the operations added in the todolist. But it's doable.
It's interesting mainly for 'big' writes

Issue Links

• is blocked by: HBASE-8380 NPE in HBaseClient$Connection.readResponse
• is depended upon by: HBASE-8338 Latency Resilience; umbrella list of issues that will help us ride over bad disk, bad region, ec2, etc.
• relates to: HBASE-5843 Improve HBase MTTR - Mean Time To Recover