[HBASE-15984] Given failure to parse a given WAL that was closed cleanly, replay the WAL.

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.3.0, 1.1.7, 0.98.23, 1.2.4, 2.0.0
• Component/s: Replication

Release Note

In some particular deployments, the Replication code believes it has
reached EOF for a WAL prior to successfully parsing all bytes known to
exist in a cleanly closed file.

If an EOF is detected due to parsing or other errors while there are still unparsed bytes before the end-of-file trailer, we now reset the WAL to the very beginning and attempt a clean read-through. Because we will retry these failures indefinitely, two additional changes are made to help with diagnostics:

* On each retry attempt, a log message like the below will be emitted at the WARN level:
    
      Processing end of WAL file '{}'. At position {}, which is too far away
      from reported file length {}. Restarting WAL reading (see HBASE-15983
      for details).

* additional metrics measure the use of this recovery mechanism. they are described in the reference guide.

• Assignee: Sean Busbey
• Reporter: Sean Busbey
• Created: 07/Jun/16 20:57
• Updated: 09/Nov/17 01:38
• Resolved: 07/Oct/16 22:36

Description

subtask for a general work around for "underlying reader failed / is in a bad state" just for the case where a WAL 1) was closed cleanly and 2) we can tell that our current offset ought not be the end of parseable entries.