[HBASE-21806] add an option to roll WAL on very slow syncs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1

Release Note

HBase RS will now roll the WAL file upon a slow sync call. hbase.regionserver.hlog.roll.on.sync.ms config has been added to HBase, with the default value of 10000 (10 seconds); if a sync of the WAL takes longer than that the file will be replaced under the assumption that we will get a faster datanode pipeline for a new file. This setting works with an existing timeout setting that will still kick in on very long syncs (5 minutes by default) and fail the operation.

• Assignee: Sergey Shelukhin
• Reporter: Sergey Shelukhin
• Created: 29/Jan/19 21:41
• Updated: 29/Apr/19 22:56
• Resolved: 31/Jan/19 00:48

Description

In large heterogeneous clusters sometimes a slow datanode can cause WAL syncs to be very slow. In this case, before the bad datanode recovers, or is discovered and repaired, it would be helpful to roll WAL on a very slow sync to get a new pipeline.
Otherwise the slow WAL will impact write latency for a long time (slow writes result in less writes result in the WAL not being rolled for longer)

Issue Links

• relates to: HBASE-22301 Consider rolling the WAL if the HDFS write pipeline is slow