[HBASE-27230] RegionServer should be aborted when WAL.sync throws TimeoutIOException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-4
• Fix Version/s: 2.6.0, 3.0.0-alpha-4
• Component/s: wal
• Labels: pull-request-available

Release Note

This changes add additional logic for WAL.sync:
If WAL.sync get a timeout exception, we wrap TimeoutIOException as a special WALSyncTimeoutIOException. When upper layer such as HRegion.doMiniBatchMutate called by HRegion.batchMutation catches this special exception, we abort the region server.

• Assignee: chenglei
• Reporter: chenglei
• Created: 21/Jul/22 03:15
• Updated: 28/Aug/24 09:01
• Resolved: 07/Mar/24 15:12

Description

As HBASE-27223 said, if WAL.sync get a timeout exception, we should abort the region server, as the design of WAL sync, is to succeed or die, there is no 'failure'. It is usually not a big deal is because we set a very large default value(5 minutes) for AbstractFSWAL.WAL_SYNC_TIMEOUT_MS, usually the WAL system will abort the region server if it can not finish the sync within 5 minutes.

In the PR, only the WAL.sync timeout in HRegion#doWALAppend ,regionServer is always aborted. For WALUtil.writeMarker, it is just record the internal state and seems it is no need to always abort the regionServer when WAL.sync timeout，it is the internal state transition that determines whether regionServer is aborted.

Issue Links

• relates to: HBASE-28803 HBase Master stuck due to improper handling of WALSyncTimeoutException within UncheckedIOException
• links to: GitHub Pull Request #4641
• links to: GitHub Pull Request #5711