[IGNITE-20116] Linearize storage updates with safeTime adjustment rules

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 3.0
• Labels: ignite-3 transactions
• Assignee: Alexander Lapin
• Reporter: Alexander Lapin
• Reviewer: Roman Puchkovskiy
• Created: 01/Aug/23 14:09
• Updated: 04/Sep/24 16:10
• Resolved: 17/Oct/23 05:58

Description

Motivation

The logic of setting safeTime explicitly prohibits setting a larger time ahead of a smaller one. In other words, all data updates within storages should be strictly ordered by the safeTime associated with such updates. Currently it's not true:

• We associate update and safe time during update command creation (see org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener)

  UpdateCommandBuilder bldr = MSG_FACTORY.updateCommand()
                   ...
                  .safeTimeLong(hybridClock.nowLong());   

• However, neither applying a given command locally nor sending it to the raft isn't linearized with associated safeTime value. In other words, it's possible that we will assign t0 to the cmd0 and t1 to the cmd1 but will apply cmd1 prior to cmd0 locally.

Simply speaking, we lack some sort of synchronization here.

Definition of Done

• It's required to linearize updates application to preserve guarantees of the monotonicity of a safeTime's adjustment.

Implementation Notes

Different options are possible:

1. We may reject a command that is associated with safeTime < already applied one. Such approach requires

1. 1. To resend the command with new safeTime in case of 1pc.

1. 1. Adjust local safeTime, and resend command with new safe time in case of 2pc.

1. Add proper synchronization both on client and server side.

1. Send pending safeTime instances with each command. More details below:

Let’s assume that there were two updateCommands cmd1(safeTime: t1) and cmd2(safeTime: t2). Let’s also assume that cmd2 was send prior to cmd2 (meaning that it was reordered). In that case, assuming that cmd2 has both t1 and t2 within its data bag, it will wait for cmd1 to bring it data in a queue or formally it will wait previous commands to apply themselves.

 

Issue Links

• causes: IGNITE-20716 Partial data loss after node restart
• causes: IGNITE-20577 Partial data loss after node restart
• causes: IGNITE-20441 ItRebalanceRecoveryTest is flaky
• relates to: IGNITE-20834 SQL query may hang forerver after node restart
• links to: GitHub Pull Request #2689