[HBASE-3136] Stale reads from ZK can break the atomic CAS operations we have in ZKAssign

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.89.20100621, 0.89.20100924, 0.90.0
• Fix Version/s: 0.90.0
• Component/s: Zookeeper
• Hadoop Flags: Reviewed
• Assignee: Jonathan Gray
• Reporter: Jonathan Gray
• Created: 20/Oct/10 23:19
• Updated: 21/Jun/20 22:35
• Resolved: 24/Oct/10 22:33

Description

With ZK based region transitions, we rely on atomic state changes of regions in transition. For example, an RS needs to atomically switch a node from OFFLINE to OPENING, or the master needs to delete nodes that are in OPENED state, etc...

The way we implement this is by:

• Read existing data (returns byte[] and version in Stat)
• Verify data is in expected state
• Update to the new state, passing the expected version previously read

This doesn't always work as expected because that initial read of the existing data could be a stale read (in ZK, writes are quorum writes but reads are not so you can get stale data).

Can provide a more explicit example if anyone is interested, but a fix is coming.

Issue Links

• breaks: HBASE-24603 Zookeeper sync() call is async
• is related to: HBASE-3137 Optimize CAS operations in ZKAssign by being optimistic rather than always doing a sync()