[PHOENIX-5266] Client can only write on Index Table and skip data table if failure happens because of region split/move etc

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 4.15.0, 4.14.1, 5.1.0, 4.14.2
• Fix Version/s: 4.15.0, 5.1.0, 4.14.2, 5.0.1
• Assignee: Mihir Monani
• Reporter: Mihir Monani
• Created: 02/May/19 10:32
• Updated: 30/May/19 18:40
• Resolved: 10/May/19 08:39

Description

With Phoenix 4.14.1 client, There is a scenario where client would skip data table write but do successful index table write. In this case, we should treat it as Data loss scenario.

 

Relevant code path :-

https://github.com/apache/phoenix/blob/4.x-HBase-1.3/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java#L994-L1043

https://github.com/apache/phoenix/blob/4.x-HBase-1.3/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java#L1089-L1109

 

Here is what happens :-

• Consider below assumptions for scenario :- 
  • max no row in single batch = 100
  • max size of batch = 2 MB
• When client faces SQLException Code 1121, it sets variable shouldRetryIndexedMutation=true.
• In scenarios where client sends batch of 100 rows only as per configuration, but batch size is >2 MB, MutationState.java#991 will split this 100 row batch into multiple smaller batches which are <2MB.
  • MutationState.java#991 :- https://github.com/apache/phoenix/blob/4.x-HBase-1.3/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java#L991
• Suppose there are 5 batches of 20 rows but client faces 1121 SQLExceptionCode on 2nd batch , then it will set shouldRetryIndexedMutation=true and it will retry all 5 batches again with only Index updates. This will results in rows missing from Data table.