[HDDS-7374] [snapshot] OM Process exiting on idle cluster

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Not A Bug
• Affects Version/s: 1.3.0
• Component/s: Ozone Manager
• Labels: ozone-snapshot

Environment

docker-compose on HDDS-6517-Snapshot

• Assignee: Hemant Kumar
• Reporter: Jyotirmoy Sinha
• Created: 21/Oct/22 09:58
• Updated: 16/Nov/22 19:29
• Resolved: 16/Nov/22 19:29

Description

Snapshots were created on the cluster, after which the system was kept idle. After certain interval the OM Process exited and shut down.

Docker logs -

2022-10-20 12:53:31,082 [org.apache.ratis.util.JvmPauseMonitor$$Lambda$398/0x00000008404cac40@bfe47a8] WARN util.JvmPauseMonitor: JvmPauseMonitor-om1: Detected pause in JVM or host machine (eg GC): pause of approximately 7196529648812ns. No GCs detected.
2022-10-20 12:53:27,062 [org.apache.ratis.util.JvmPauseMonitor$$Lambda$398/0x00000008404cac40@bfe47a8] INFO server.RaftServer: om1: close
2022-10-20 12:53:27,128 [org.apache.hadoop.util.JvmPauseMonitor$Monitor@60844992] WARN util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 7197056ms
No GCs detected
2022-10-20 12:53:27,166 [org.apache.ratis.util.JvmPauseMonitor$$Lambda$398/0x00000008404cac40@bfe47a8] INFO server.RaftServer$Division: om1@group-C5BA1605619E: shutdown
2022-10-20 12:53:27,177 [org.apache.ratis.util.JvmPauseMonitor$$Lambda$398/0x00000008404cac40@bfe47a8] INFO util.JmxRegister: Successfully un-registered JMX Bean with object name Ratis:service=RaftServer,group=group-C5BA1605619E,id=om1
2022-10-20 12:53:27,196 [org.apache.ratis.util.JvmPauseMonitor$$Lambda$398/0x00000008404cac40@bfe47a8] INFO impl.RoleInfo: om1: shutdown om1@group-C5BA1605619E-LeaderStateImpl
2022-10-20 12:53:27,212 [org.apache.ratis.util.JvmPauseMonitor$$Lambda$398/0x00000008404cac40@bfe47a8] INFO impl.PendingRequests: om1@group-C5BA1605619E-PendingRequests: sendNotLeaderResponses
2022-10-20 12:53:27,366 [org.apache.ratis.util.JvmPauseMonitor$$Lambda$398/0x00000008404cac40@bfe47a8] INFO impl.StateMachineUpdater: om1@group-C5BA1605619E-StateMachineUpdater: set stopIndex = 10
2022-10-20 12:53:27,368 [om1@group-C5BA1605619E-StateMachineUpdater] INFO ratis.OzoneManagerStateMachine: Current Snapshot Index (t:1, i:10)
2022-10-20 12:53:27,390 [om1@group-C5BA1605619E-StateMachineUpdater] INFO impl.StateMachineUpdater: om1@group-C5BA1605619E-StateMachineUpdater: Took a snapshot at index 10
2022-10-20 12:53:27,398 [om1@group-C5BA1605619E-StateMachineUpdater] INFO impl.StateMachineUpdater: om1@group-C5BA1605619E-StateMachineUpdater: snapshotIndex: updateIncreasingly -1 -> 10
2022-10-20 12:53:27,398 [om1@group-C5BA1605619E-StateMachineUpdater] INFO ratis.OzoneManagerStateMachine: StateMachine has shutdown. Shutdown OzoneManager if not already shutdown.
2022-10-20 12:53:27,399 [om1@group-C5BA1605619E-StateMachineUpdater] INFO om.OzoneManager: om1[om:9862]: Stopping Ozone Manager
2022-10-20 12:53:27,474 [om1@group-C5BA1605619E-StateMachineUpdater] INFO ipc.Server: Stopping server on 9862
2022-10-20 12:53:27,609 [IPC Server listener on 9862] INFO ipc.Server: Stopping IPC Server listener on 9862
2022-10-20 12:53:27,628 [IPC Server Responder] INFO ipc.Server: Stopping IPC Server Responder