[HBASE-7268] correct local region location cache information can be overwritten (or deleted) w/stale information from an old server

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.95.2
• Fix Version/s: 0.95.0
• Hadoop Flags: Incompatible change, Reviewed

Release Note

On region open, save the edit seqId and then write it into META. On region move, the region is opened again with a greater seqId somewhere else. Pass the client the seqid to the client when asks about locations. Client can reason about cache invalidation with seqids (if seqid for new location is < its current seqid, it can recognize the new location stale).

• Assignee: Sergey Shelukhin
• Reporter: Sergey Shelukhin
• Created: 04/Dec/12 03:25
• Updated: 23/Sep/13 18:31
• Resolved: 24/Jan/13 17:21

Description

Discovered via HBASE-7250; related to HBASE-5877.
Test is writing from multiple threads.
Server A has region R; client knows that.
R gets moved from A to server B.
B gets killed.
R gets moved by master to server C.
~15 seconds later, client tries to write to it (on A?).
Multiple client threads report from RegionMoved exception processing logic "R moved from C to B", even though such transition never happened (neither in nor before the sequence described below). Not quite sure how the client learned of the transition to C, I assume it's from meta from some other thread...
Then, put fails (it may fail due to accumulated errors that are not logged, which I am investigating... but the bogus cache update is there nonwithstanding).

I have a patch but not sure if it works, test still fails locally for yet unknown reason.

Issue Links

• relates to: HBASE-7520 org.apache.hadoop.hbase.IntegrationTestRebalanceAndKillServersTargeted fails when I cd hbase-it and mvn verify
• relates to: HBASE-7638 [0.94] region cache entry should only be removed on error if the error is from the server currently in cache