[HDFS-12639] BPOfferService lock may stall all service actors

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.8.0
• Component/s: datanode
• Assignee: Hanisha Koneru
• Reporter: Daryn Sharp
• Created: 11/Oct/17 19:23
• Updated: 29/Jan/23 09:39

Description

BPOfferService manages BPServiceActor instances for the active and standby. It uses a RW lock to primarily protect registration information while determining the active/standby from heartbeats.

Unfortunately the write lock is held during command processing. If an actor is experiencing high latency processing commands, the other actor will neither be able to register (blocked in createRegistration, setNamespaceInfo, verifyAndSetNamespaceInfo) nor process heartbeats (blocked in updateActorStatesFromHeartbeat).

The worst case scenario for processing commands while holding the lock is re-registration. The actor will loop, catching and logging exceptions, leaving the other actor blocked for an non-deterministic (possibly infinite) amount of time.

The lock must not be held during command processing.

Issue Links

• relates to: HDFS-12645 FSDatasetImpl lock will stall BP service actors and may cause missing blocks