[HBASE-21564] race condition in WAL rolling resulting in size-based rolling getting stuck

• Type: Bug
• Status: Patch Available
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 3.0.0-alpha-1, 2.2.0
• Component/s: asyncclient, wal
• Assignee: Sergey Shelukhin
• Reporter: Sergey Shelukhin
• Created: 07/Dec/18 02:43
• Updated: 22/Jun/20 12:27

Description

Manifests at least with AsyncFsWriter.
There's a window after LogRoller replaces the writer in the WAL, but before it sets the rollLog boolean to false in the finally, where the WAL class can request another log roll (it can happen in particular when the logs are getting archived in the LogRoller thread, and there's high write volume causing the logs to roll quickly).
LogRoller will blindly reset the rollLog flag in finally and "forget" about this request.
AsyncWAL in turn never requests it again because its own rollRequested field is set and it expects a callback. Logs don't get rolled until a periodic roll is triggered after that.

The acknowledgment of roll requests by LogRoller should be atomic.

Issue Links

• links to: RB