[2023-05-24 17:47:36,114] INFO [LocalLog partition=__consumer_offsets-40, dir=/opt/mdmadm/kafka/kafka_2.13-3.3.1/./notification] Rolled new log segment at offset 7854 in 4 ms. (kafka.log.LocalLog)
[2023-05-24 17:47:36,114] INFO [LocalLog partition=__consumer_offsets-49, dir=/opt/mdmadm/kafka/kafka_2.13-3.3.1/./notification] Rolled new log segment at offset 7741 in 3 ms. (kafka.log.LocalLog)
[2023-05-24 17:47:36,118] INFO [ProducerStateManager partition=__consumer_offsets-49] Wrote producer snapshot at offset 7741 with 0 producer ids in 3 ms. (kafka.log.ProducerStateManager)
[2023-05-24 17:47:36,119] INFO [ProducerStateManager partition=__consumer_offsets-40] Wrote producer snapshot at offset 7854 with 0 producer ids in 4 ms. (kafka.log.ProducerStateManager)
[2023-05-24 17:47:41,518] INFO [RaftManager nodeId=1] Completed transition to Unattached(epoch=58254, voters=[1, 2, 3], electionTimeoutMs=1505) (org.apache.kafka.raft.QuorumState)
[2023-05-24 17:47:41,518] INFO [Controller 1] In the new epoch 58254, the leader is (none). (org.apache.kafka.controller.QuorumController)
[2023-05-24 17:47:41,518] INFO [RaftManager nodeId=1] Vote request VoteRequestData(clusterId='_MCG9doEQZ6-kM2fjohf4Q', topics=[TopicData(topicName='__cluster_metadata', partitions=[PartitionData(partitionIndex=0, candidateEpoch=58254, candidateId=3, lastOffsetEpoch=58253, lastOffset=6114859)])]) with epoch 58254 is rejected (org.apache.kafka.raft.KafkaRaftClient)
[2023-05-24 17:47:41,635] INFO [RaftManager nodeId=1] Completed transition to Unattached(epoch=58255, voters=[1, 2, 3], electionTimeoutMs=1388) (org.apache.kafka.raft.QuorumState)
[2023-05-24 17:47:41,635] INFO [RaftManager nodeId=1] Vote request VoteRequestData(clusterId='_MCG9doEQZ6-kM2fjohf4Q', topics=[TopicData(topicName='__cluster_metadata', partitions=[PartitionData(partitionIndex=0, candidateEpoch=58255, candidateId=3, lastOffsetEpoch=58253, lastOffset=6114859)])]) with epoch 58255 is rejected (org.apache.kafka.raft.KafkaRaftClient)
[2023-05-24 17:47:41,635] INFO [Controller 1] In the new epoch 58255, the leader is (none). (org.apache.kafka.controller.QuorumController)
[2023-05-24 17:47:41,644] INFO [RaftManager nodeId=1] Completed transition to Unattached(epoch=58256, voters=[1, 2, 3], electionTimeoutMs=1378) (org.apache.kafka.raft.QuorumState)
[2023-05-24 17:47:41,644] INFO [RaftManager nodeId=1] Vote request VoteRequestData(clusterId='_MCG9doEQZ6-kM2fjohf4Q', topics=[TopicData(topicName='__cluster_metadata', partitions=[PartitionData(partitionIndex=0, candidateEpoch=58256, candidateId=3, lastOffsetEpoch=58253, lastOffset=6114859)])]) with epoch 58256 is rejected (org.apache.kafka.raft.KafkaRaftClient)
[2023-05-24 17:47:41,644] INFO [Controller 1] In the new epoch 58256, the leader is (none). (org.apache.kafka.controller.QuorumController)
[2023-05-24 17:47:42,156] INFO [RaftManager nodeId=1] Completed transition to Unattached(epoch=58257, voters=[1, 2, 3], electionTimeoutMs=866) (org.apache.kafka.raft.QuorumState)
[2023-05-24 17:47:42,156] INFO [RaftManager nodeId=1] Vote request VoteRequestData(clusterId='_MCG9doEQZ6-kM2fjohf4Q', topics=[TopicData(topicName='__cluster_metadata', partitions=[PartitionData(partitionIndex=0, candidateEpoch=58257, candidateId=3, lastOffsetEpoch=58253, lastOffset=6114859)])]) with epoch 58257 is rejected (org.apache.kafka.raft.KafkaRaftClient)
[2023-05-24 17:47:42,156] INFO [Controller 1] In the new epoch 58257, the leader is (none). (org.apache.kafka.controller.QuorumController)
[2023-05-24 17:47:42,268] INFO [RaftManager nodeId=1] Completed transition to Unattached(epoch=58258, voters=[1, 2, 3], electionTimeoutMs=754) (org.apache.kafka.raft.QuorumState)
[2023-05-24 17:47:42,268] INFO [RaftManager nodeId=1] Vote request VoteRequestData(clusterId='_MCG9doEQZ6-kM2fjohf4Q', topics=[TopicData(topicName='__cluster_metadata', partitions=[PartitionData(partitionIndex=0, candidateEpoch=58258, candidateId=3, lastOffsetEpoch=58253, lastOffset=6114859)])]) with epoch 58258 is rejected (org.apache.kafka.raft.KafkaRaftClient)
[2023-05-24 17:47:42,268] INFO [Controller 1] In the new epoch 58258, the leader is (none). (org.apache.kafka.controller.QuorumController)
[2023-05-24 17:47:42,679] INFO [RaftManager nodeId=1] Completed transition to Unattached(epoch=58259, voters=[1, 2, 3], electionTimeoutMs=344) (org.apache.kafka.raft.QuorumState)
[2023-05-24 17:47:42,679] INFO [RaftManager nodeId=1] Vote request VoteRequestData(clusterId='_MCG9doEQZ6-kM2fjohf4Q', topics=[TopicData(topicName='__cluster_metadata', partitions=[PartitionData(partitionIndex=0, candidateEpoch=58259, candidateId=3, lastOffsetEpoch=58253, lastOffset=6114859)])]) with epoch 58259 is rejected (org.apache.kafka.raft.KafkaRaftClient)
[2023-05-24 17:47:42,679] INFO [Controller 1] In the new epoch 58259, the leader is (none). (org.apache.kafka.controller.QuorumController)
[2023-05-24 17:47:42,841] INFO [RaftManager nodeId=1] Completed transition to Unattached(epoch=58260, voters=[1, 2, 3], electionTimeoutMs=181) (org.apache.kafka.raft.QuorumState)
[2023-05-24 17:47:42,841] INFO [Controller 1] In the new epoch 58260, the leader is (none). (org.apache.kafka.controller.QuorumController)
[2023-05-24 17:47:42,845] INFO [RaftManager nodeId=1] Completed transition to Voted(epoch=58260, votedId=2, voters=[1, 2, 3], electionTimeoutMs=1437) (org.apache.kafka.raft.QuorumState)
[2023-05-24 17:47:42,845] INFO [RaftManager nodeId=1] Vote request VoteRequestData(clusterId='_MCG9doEQZ6-kM2fjohf4Q', topics=[TopicData(topicName='__cluster_metadata', partitions=[PartitionData(partitionIndex=0, candidateEpoch=58260, candidateId=2, lastOffsetEpoch=58253, lastOffset=6114863)])]) with epoch 58260 is granted (org.apache.kafka.raft.KafkaRaftClient)
[2023-05-24 17:47:42,860] INFO [RaftManager nodeId=1] Completed transition to FollowerState(fetchTimeoutMs=2000, epoch=58260, leaderId=2, voters=[1, 2, 3], highWatermark=Optional[LogOffsetMetadata(offset=6114862, metadata=Optional.empty)], fetchingSnapshot=Optional.empty) (org.apache.kafka.raft.QuorumState)
[2023-05-24 17:47:42,860] INFO [Controller 1] In the new epoch 58260, the leader is 2. (org.apache.kafka.controller.QuorumController)
[2023-05-24 17:47:49,635] ERROR Failed to clean up log for __consumer_offsets-49 in dir /opt/mdmadm/kafka/kafka_2.13-3.3.1/./notification due to IOException (kafka.server.LogDirFailureChannel)
java.nio.file.NoSuchFileException: /opt/mdmadm/kafka/kafka_2.13-3.3.1/./notification/__consumer_offsets-49/00000000000000004146.log
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:935)
	at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:248)
	at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:491)
	at kafka.log.LocalLog$.$anonfun$deleteSegmentFiles$1(LocalLog.scala:989)
	at kafka.log.LocalLog$.$anonfun$deleteSegmentFiles$1$adapted(LocalLog.scala:987)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at kafka.log.LocalLog$.deleteSegmentFiles(LocalLog.scala:987)
	at kafka.log.LocalLog$.$anonfun$replaceSegments$7(LocalLog.scala:950)
	at scala.collection.immutable.List.map(List.scala:246)
	at scala.collection.immutable.List.map(List.scala:79)
	at kafka.log.LocalLog$.replaceSegments(LocalLog.scala:938)
	at kafka.log.UnifiedLog.replaceSegments(UnifiedLog.scala:1972)
	at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:626)
	at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:546)
	at kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:545)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at kafka.log.Cleaner.doClean(LogCleaner.scala:545)
	at kafka.log.Cleaner.clean(LogCleaner.scala:510)
	at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:394)
	at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:366)
	at kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:345)
	at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:332)
	at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
	Suppressed: java.nio.file.NoSuchFileException: /opt/mdmadm/kafka/kafka_2.13-3.3.1/./notification/__consumer_offsets-49/00000000000000004146.log -> /opt/mdmadm/kafka/kafka_2.13-3.3.1/./notification/__consumer_offsets-49/00000000000000004146.log.deleted
		at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
		at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
		at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396)
		at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
		at java.nio.file.Files.move(Files.java:1395)
		at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:932)
		... 22 more
[2023-05-24 17:47:49,638] WARN [ReplicaManager broker=1] Stopping serving replicas in dir /opt/mdmadm/kafka/kafka_2.13-3.3.1/./notification (kafka.server.ReplicaManager)
[2023-05-24 17:47:49,644] INFO [ReplicaFetcherManager on broker 1] Removed fetcher for partitions HashSet(__consumer_offsets-22, __consumer_offsets-30, __consumer_offsets-25, __consumer_offsets-35, __consumer_offsets-37, mdmnxst-topic-0, __consumer_offsets-38, __consumer_offsets-13, __consumer_offsets-8, __consumer_offsets-21, __consumer_offsets-4, __consumer_offsets-27, __consumer_offsets-7, __consumer_offsets-9, __consumer_offsets-46, __consumer_offsets-41, __consumer_offsets-33, __consumer_offsets-23, __consumer_offsets-49, __consumer_offsets-47, __consumer_offsets-16, __consumer_offsets-28, __consumer_offsets-31, __consumer_offsets-36, __consumer_offsets-42, __consumer_offsets-3, __consumer_offsets-18, __consumer_offsets-15, __consumer_offsets-24, __consumer_offsets-17, __consumer_offsets-48, __consumer_offsets-19, __consumer_offsets-11, __consumer_offsets-2, __consumer_offsets-43, __consumer_offsets-6, __consumer_offsets-14, __consumer_offsets-20, __consumer_offsets-0, __consumer_offsets-44, __consumer_offsets-39, __consumer_offsets-12, __consumer_offsets-45, __consumer_offsets-1, __consumer_offsets-5, __consumer_offsets-26, __consumer_offsets-29, __consumer_offsets-34, __consumer_offsets-10, __consumer_offsets-32, __consumer_offsets-40) (kafka.server.ReplicaFetcherManager)
[2023-05-24 17:47:49,645] INFO [ReplicaAlterLogDirsManager on broker 1] Removed fetcher for partitions HashSet(__consumer_offsets-22, __consumer_offsets-30, __consumer_offsets-25, __consumer_offsets-35, __consumer_offsets-37, mdmnxst-topic-0, __consumer_offsets-38, __consumer_offsets-13, __consumer_offsets-8, __consumer_offsets-21, __consumer_offsets-4, __consumer_offsets-27, __consumer_offsets-7, __consumer_offsets-9, __consumer_offsets-46, __consumer_offsets-41, __consumer_offsets-33, __consumer_offsets-23, __consumer_offsets-49, __consumer_offsets-47, __consumer_offsets-16, __consumer_offsets-28, __consumer_offsets-31, __consumer_offsets-36, __consumer_offsets-42, __consumer_offsets-3, __consumer_offsets-18, __consumer_offsets-15, __consumer_offsets-24, __consumer_offsets-17, __consumer_offsets-48, __consumer_offsets-19, __consumer_offsets-11, __consumer_offsets-2, __consumer_offsets-43, __consumer_offsets-6, __consumer_offsets-14, __consumer_offsets-20, __consumer_offsets-0, __consumer_offsets-44, __consumer_offsets-39, __consumer_offsets-12, __consumer_offsets-45, __consumer_offsets-1, __consumer_offsets-5, __consumer_offsets-26, __consumer_offsets-29, __consumer_offsets-34, __consumer_offsets-10, __consumer_offsets-32, __consumer_offsets-40) (kafka.server.ReplicaAlterLogDirsManager)
[2023-05-24 17:47:49,654] WARN [ReplicaManager broker=1] Broker 1 stopped fetcher for partitions __consumer_offsets-22,__consumer_offsets-30,__consumer_offsets-25,__consumer_offsets-35,__consumer_offsets-37,mdmnxst-topic-0,__consumer_offsets-38,__consumer_offsets-13,__consumer_offsets-8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,__consumer_offsets-41,__consumer_offsets-33,__consumer_offsets-23,__consumer_offsets-49,__consumer_offsets-47,__consumer_offsets-16,__consumer_offsets-28,__consumer_offsets-31,__consumer_offsets-36,__consumer_offsets-42,__consumer_offsets-3,__consumer_offsets-18,__consumer_offsets-15,__consumer_offsets-24,__consumer_offsets-17,__consumer_offsets-48,__consumer_offsets-19,__consumer_offsets-11,__consumer_offsets-2,__consumer_offsets-43,__consumer_offsets-6,__consumer_offsets-14,__consumer_offsets-20,__consumer_offsets-0,__consumer_offsets-44,__consumer_offsets-39,__consumer_offsets-12,__consumer_offsets-45,__consumer_offsets-1,__consumer_offsets-5,__consumer_offsets-26,__consumer_offsets-29,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,__consumer_offsets-40 and stopped moving logs for partitions  because they are in the failed log directory /opt/mdmadm/kafka/kafka_2.13-3.3.1/./notification. (kafka.server.ReplicaManager)
[2023-05-24 17:47:49,654] WARN Stopping serving logs in dir /opt/mdmadm/kafka/kafka_2.13-3.3.1/./notification (kafka.log.LogManager)
[2023-05-24 17:47:49,656] ERROR Shutdown broker because all log dirs in /opt/mdmadm/kafka/kafka_2.13-3.3.1/./notification have failed (kafka.log.LogManager)