[HBASE-14317] Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.2.0, 1.1.1
• Fix Version/s: 1.2.0, 1.3.0, 2.0.0
• Component/s: wal
• Hadoop Flags: Reviewed

Release Note

Tighten up WAL-use semantic.

1. If an append or a sync throws an exception, all subsequent attempts at using the log will also throw this same exception. The WAL is now a lame-duck until you roll it.
2. If a successful append, and then we fail to sync the append, this is a fatal exception. The container must abort to replay the WAL logs even though we have told the client that the appends failed.

The above rules have been applied laxly up to this; it used to be possible to get a good sync to go in over the top of a failed append. This has been fixed in this patch.

Also fixed a hang in the WAL subsystem if a request to pause the write pipeline took on a failed sync. before the roll requests sync got scheduled.


TODO: Revisit our WAL system. HBASE-12751 helps rationalize our write pipeline. In particular, it manages sequenceid inside mvcc which should make it so we can purge mechanism that writes empty, unflushed appends just to get the next sequenceid... problematic when WAL goes lame-duck. Lets get it in.
TODO: A successful append followed by a failed sync probably only needs us replace the WAL (if we have signalled the client that the appends failed). Bummer is that replicating, these last appends might make it to the sink cluster or get replayed during recovery. HBase should keep its own WAL length? Or sequenceid of last successful sync should be passed when doing recovery and replication?

• Assignee: Michael Stack
• Reporter: Michael Stack
• Created: 26/Aug/15 18:46
• Updated: 01/Jul/22 21:05
• Resolved: 07/Sep/15 04:55

Description

hbase-1.1.1 and hadoop-2.7.1

We try to roll logs because can't append (See HDFS-8960) but we get stuck. See attached thread dump and associated log. What is interesting is that syncers are waiting to take syncs to run and at same time we want to flush so we are waiting on a safe point but there seems to be nothing in our ring buffer; did we go to roll log and not add safe point sync to clear out ringbuffer?

Needs a bit of study. Try to reproduce.

Issue Links

• is related to: HDFS-8960 DFS client says "no more good datanodes being available to try" on a single drive failure
• relates to: HBASE-13971 Flushes stuck since 6 hours on a regionserver.
• relates to: HBASE-16960 RegionServer hang when aborting

Sub-Tasks

• New TestWALLockup broken by addendum added to parent issue: Closed
• Backport parent 'HBASE-14317 Stuck FSHLog' issue to 1.1: Closed
• Get TestAccessController* passing again on branch-1: Closed
• Stamp failed appends with sequenceid too.... Cleans up latches: Closed