[RATIS-2055] Changed logic of LeaderState.isReady impacts application

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: Leader
• Assignee: Janus Chow
• Reporter: Duong
• Created: 12/Apr/24 00:25
• Updated: 19/Apr/24 15:21
• Resolved: 19/Apr/24 15:21

Description

RATIS-2019 changes the logic of `LeaderState.isReady`. Before, isReady itself performs the logic to check if the leader state is ready. After the change, the logic is moved to `LeaderState.checkReady` and `isReady` now only reads the result.

This change impacts client applications that rely on `isReady` check. For example, Ozone's SCMStateMachine listens to notifyTermIndexUpdated and checks if the leader state is ready. However, because the `leader.checkReady` is called after the `notifyTermIndexUpdated`, SCMStateMachine never sees leader ready state in the intercepted `notifyTermIndexUpdated` events.

RaftServerImpl.applyLogToStateMachine

CompletableFuture<Message> applyLogToStateMachine(ReferenceCountedObject<LogEntryProto> nextRef)
    throws RaftLogIOException {
  LogEntryProto next = nextRef.get();
  if (!next.hasStateMachineLogEntry()) {
    stateMachine.event().notifyTermIndexUpdated(next.getTerm(), next.getIndex());
  }

  if (next.hasConfigurationEntry()) {
    // the reply should have already been set. only need to record
    // the new conf in the metadata file and notify the StateMachine.
    state.writeRaftConfiguration(next);
    stateMachine.event().notifyConfigurationChanged(next.getTerm(), next.getIndex(), next.getConfigurationEntry());
    role.getLeaderState().ifPresent(leader -> leader.checkReady(next)); 
....

 

Issue Links

• is caused by: RATIS-2019 Fixed abnormal exit of StateMachineUpdater
• is related to: HDDS-10690 SCMStateMachine Override LeaderEventApi.notifyLeaderReady
• links to: GitHub Pull Request #1068