[ACCUMULO-1083] add concurrency to HDFS write-ahead log

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: tserver
• Assignee: Unassigned
• Reporter: Adam Fuchs
• Created: 22/Feb/13 02:53
• Updated: 04/Apr/15 02:28
• Resolved: 04/Apr/15 02:28

Description

When running tablet servers on beefy nodes (lots of disks), the write-ahead log can be a serious bottleneck. Today we ran a continuous ingest test of 1.5-SNAPSHOT on an 8-node (plus a master node) cluster in which the nodes had 32 cores and 15 drives each. Running with write-ahead log off resulted in a >4x performance improvement sustained over a long period.

I believe the culprit is that the WAL is only using one file at a time per tablet server, which means HDFS is only appending to one drive (plus replicas). If we increase the number of concurrent WAL files supported on a tablet server we could probably drastically improve the performance on systems with many disks. As it stands, I believe Accumulo is significantly more optimized for a larger number of smaller nodes (3-4 drives).

Issue Links

• is related to: ACCUMULO-1085 make the number of threads for assignment configurable
• is related to: ACCUMULO-1754 support scale-up behavior in BatchWriter
• is related to: ACCUMULO-1177 Decrease time it takes to recover after tablet server failures
• is related to: HBASE-5699 Run with > 1 WAL in HRegionServer