[HDDS-10595] [snapshot-LR] OM node is not shown in 'om roles' as follower node

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: OM, Snapshot
• Labels: ozone-snapshot
• Assignee: Unassigned
• Reporter: Jyotirmoy Sinha
• Created: 27/Mar/24 06:51
• Updated: 27/Mar/24 06:51

Description

OM node is not shown in 'om roles' as follower node

OM roles - CLI -

[root@node~]# ozone admin om roles -id=ozone1709211738
om233 : FOLLOWER (node1.domain.com)
om232 : LEADER (node2.domain.com) 

OM log output -

2024-03-19 00:00:01,203 INFO [om363-server-thread17]-org.apache.ratis.server.RaftServer$Division: om363@group-A826AF593A36: Failed appendEntries as snapshot (30450723) installation is in progress
2024-03-19 00:00:01,203 INFO [om363-server-thread17]-org.apache.ratis.server.RaftServer$Division: om363@group-A826AF593A36: inconsistency entries. Reply:om232<-om363#14732148:FAIL-t54,INCONSISTENCY,nextIndex=0,followerCommit=-1,matchIndex=-1
2024-03-19 00:00:01,203 INFO [grpc-default-executor-6]-org.apache.ratis.server.impl.SnapshotInstallationHandler: om363@group-A826AF593A36: receive installSnapshot: om232->om234#0-t54,notify:(t:52, i:30450723)
2024-03-19 00:00:01,203 INFO [grpc-default-executor-6]-org.apache.ratis.server.impl.SnapshotInstallationHandler: om363@group-A826AF593A36: reply installSnapshot: om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:00:01,203 INFO [grpc-default-executor-6]-org.apache.ratis.grpc.server.GrpcServerProtocolService: om363: Completed INSTALL_SNAPSHOT, lastRequest: om232->om234#0-t54,notify:(t:52, i:30450723)
2024-03-19 00:00:01,203 INFO [grpc-default-executor-6]-org.apache.ratis.grpc.server.GrpcServerProtocolService: om363: Completed INSTALL_SNAPSHOT, lastReply: null
2024-03-19 00:00:01,203 INFO [om363-server-thread17]-org.apache.ratis.server.RaftServer$Division: om363@group-A826AF593A36: Failed appendEntries as snapshot (30450723) installation is in progress
2024-03-19 00:00:01,203 INFO [om363-server-thread17]-org.apache.ratis.server.RaftServer$Division: om363@group-A826AF593A36: inconsistency entries. Reply:om232<-om363#14732149:FAIL-t54,INCONSISTENCY,nextIndex=0,followerCommit=-1,matchIndex=-1
2024-03-19 00:00:01,203 INFO [grpc-default-executor-6]-org.apache.ratis.server.impl.SnapshotInstallationHandler: om363@group-A826AF593A36: receive installSnapshot: om232->om234#0-t54,notify:(t:52, i:30450723)
2024-03-19 00:00:01,203 INFO [grpc-default-executor-6]-org.apache.ratis.server.impl.SnapshotInstallationHandler: om363@group-A826AF593A36: reply installSnapshot: om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:00:01,203 INFO [grpc-default-executor-6]-org.apache.ratis.grpc.server.GrpcServerProtocolService: om363: Completed INSTALL_SNAPSHOT, lastRequest: om232->om234#0-t54,notify:(t:52, i:30450723)
2024-03-19 00:00:01,203 INFO [grpc-default-executor-6]-org.apache.ratis.grpc.server.GrpcServerProtocolService: om363: Completed INSTALL_SNAPSHOT, lastReply: null
2024-03-19 00:00:01,204 INFO [om363-server-thread17]-org.apache.ratis.server.RaftServer$Division: om363@group-A826AF593A36: Failed appendEntries as snapshot (30450723) installation is in progress
2024-03-19 00:00:01,204 INFO [om363-server-thread17]-org.apache.ratis.server.RaftServer$Division: om363@group-A826AF593A36: inconsistency entries. Reply:om232<-om363#14732150:FAIL-t54,INCONSISTENCY,nextIndex=0,followerCommit=-1,matchIndex=-1
2024-03-19 00:00:01,204 INFO [grpc-default-executor-7]-org.apache.ratis.server.impl.SnapshotInstallationHandler: om363@group-A826AF593A36: receive installSnapshot: om232->om234#0-t54,notify:(t:52, i:30450723)
2024-03-19 00:00:01,204 INFO [grpc-default-executor-7]-org.apache.ratis.server.impl.SnapshotInstallationHandler: om363@group-A826AF593A36: reply installSnapshot: om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:00:01,204 INFO [grpc-default-executor-7]-org.apache.ratis.grpc.server.GrpcServerProtocolService: om363: Completed INSTALL_SNAPSHOT, lastRequest: om232->om234#0-t54,notify:(t:52, i:30450723)
2024-03-19 00:00:01,204 INFO [grpc-default-executor-7]-org.apache.ratis.grpc.server.GrpcServerProtocolService: om363: Completed INSTALL_SNAPSHOT, lastReply: null
2024-03-19 00:00:01,204 INFO [om363-server-thread17]-org.apache.ratis.server.RaftServer$Division: om363@group-A826AF593A36: Failed appendEntries as snapshot (30450723) installation is in progress 

 OM Leader log output pertaining to above OM Follower -

2024-03-19 00:03:11,531 INFO [grpc-default-executor-75]-org.apache.ratis.grpc.server.GrpcLogAppender: om232@group-A826AF593A36->om234-InstallSnapshotResponseHandler: received a reply om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:03:11,531 INFO [grpc-default-executor-75]-org.apache.ratis.grpc.server.GrpcLogAppender: om232@group-A826AF593A36->om234-InstallSnapshotResponseHandler: received a reply om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:03:11,532 INFO [grpc-default-executor-75]-org.apache.ratis.grpc.server.GrpcLogAppender: om232@group-A826AF593A36->om234-InstallSnapshotResponseHandler: received a reply om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:03:11,532 INFO [grpc-default-executor-75]-org.apache.ratis.grpc.server.GrpcLogAppender: om232@group-A826AF593A36->om234-InstallSnapshotResponseHandler: received a reply om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:03:11,533 INFO [grpc-default-executor-75]-org.apache.ratis.grpc.server.GrpcLogAppender: om232@group-A826AF593A36->om234-InstallSnapshotResponseHandler: received a reply om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:03:11,534 INFO [grpc-default-executor-26]-org.apache.ratis.grpc.server.GrpcLogAppender: om232@group-A826AF593A36->om234-InstallSnapshotResponseHandler: received a reply om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:03:11,534 INFO [grpc-default-executor-26]-org.apache.ratis.grpc.server.GrpcLogAppender: om232@group-A826AF593A36->om234-InstallSnapshotResponseHandler: received a reply om232<-om363#0:FAIL-t54,IN_PROGRESS
2024-03-19 00:03:11,535 INFO [grpc-default-executor-75]-org.apache.ratis.grpc.server.GrpcLogAppender: om232@group-A826AF593A36->om234-InstallSnapshotResponseHandler: received a reply om232<-om363#0:FAIL-t54,IN_PROGRESS 

From logs the snapshot installation is failing in the the follower OM, this log is flooding in the follower OM and similar error is present in leader indicating the snapshot-installation is failing/in-progress.