2022-08-22 17:54:43,082 INFO org.apache.kafka.connect.runtime.Worker: Stopping connector MirrorSourceConnector 2022-08-22 17:54:43,085 INFO org.apache.kafka.connect.runtime.Worker: Stopped connector MirrorSourceConnector 2022-08-22 17:54:43,085 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-14, groupId=bdkfkwx006-mm2] Finished stopping tasks in preparation for rebalance 2022-08-22 17:54:43,102 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-8, groupId=bdkfkwx006-mm2] Attempt to heartbeat failed since group is rebalancing 2022-08-22 17:54:43,120 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-14, groupId=bdkfkwx006-mm2] Finished flushing status backing store in preparation for rebalance 2022-08-22 17:54:43,120 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-14, groupId=bdkfkwx006-mm2] Joined group at generation 2 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-14-db197b17-96f4-4237-ad46-723c90444e08', leaderUrl='https://redacted-url//', offset=1157, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorHeartbeatConnector-0], revokedConnectorIds=[MirrorSourceConnector], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:43,120 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-14, groupId=bdkfkwx006-mm2] Starting connectors and tasks using config offset 1157 2022-08-22 17:54:43,120 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-14, groupId=bdkfkwx006-mm2] Finished starting connectors and tasks 2022-08-22 17:54:43,120 INFO org.apache.kafka.connect.runtime.distributed.WorkerCoordinator: [Worker clientId=connect-14, groupId=bdkfkwx006-mm2] Rebalance started 2022-08-22 17:54:43,120 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-14, groupId=bdkfkwx006-mm2] (Re-)joining group 2022-08-22 17:54:43,135 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-1, groupId=kfkhwshared00-mm2] Successfully joined group with generation 3 2022-08-22 17:54:43,135 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-1, groupId=kfkhwshared00-mm2] Joined group at generation 3 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-1-377e2e96-56a3-48ca-92b3-568b529978d8', leaderUrl='https://redacted-url-13/', offset=347, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorSourceConnector-2, MirrorHeartbeatConnector-0], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:43,135 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-1, groupId=kfkhwshared00-mm2] Starting connectors and tasks using config offset 347 2022-08-22 17:54:43,135 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-1, groupId=kfkhwshared00-mm2] Finished starting connectors and tasks 2022-08-22 17:54:43,141 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-10, groupId=bdkfkbup007-mm2] Successfully joined group with generation 3 2022-08-22 17:54:43,141 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-10, groupId=bdkfkbup007-mm2] Joined group at generation 3 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-10-d042b37e-d807-4301-b637-8e17b1094aee', leaderUrl='https://redacted-url-9/', offset=1225, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorHeartbeatConnector-0], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:43,141 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-10, groupId=bdkfkbup007-mm2] Starting connectors and tasks using config offset 1225 2022-08-22 17:54:43,142 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-10, groupId=bdkfkbup007-mm2] Finished starting connectors and tasks 2022-08-22 17:54:43,241 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-12, groupId=lkc-6k37mq-mm2] Successfully joined group with generation 6 2022-08-22 17:54:43,242 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-12, groupId=lkc-6k37mq-mm2] Joined group at generation 6 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-12-b4d9a0ac-677e-4dc4-87f4-48afa3dd9687', leaderUrl='https://redacted-url-8/', offset=947, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorHeartbeatConnector-0], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:43,242 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-12, groupId=lkc-6k37mq-mm2] Starting connectors and tasks using config offset 947 2022-08-22 17:54:43,242 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-12, groupId=lkc-6k37mq-mm2] Finished starting connectors and tasks 2022-08-22 17:54:44,034 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-19, groupId=lkc-w727wj-mm2] Successfully joined group with generation 65 2022-08-22 17:54:44,034 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-19, groupId=lkc-w727wj-mm2] Joined group at generation 65 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-19-87d31e15-6a0c-4005-a760-a4965dbc6f6a', leaderUrl='https://redacted-url-10/', offset=566, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorHeartbeatConnector-0], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:44,034 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-19, groupId=lkc-w727wj-mm2] Starting connectors and tasks using config offset 566 2022-08-22 17:54:44,035 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-19, groupId=lkc-w727wj-mm2] Finished starting connectors and tasks 2022-08-22 17:54:44,145 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-20, groupId=bdkfkbup007-mm2] Successfully joined group with generation 65 2022-08-22 17:54:44,146 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-20, groupId=bdkfkbup007-mm2] Joined group at generation 65 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-20-6999e8a4-ddf9-4cfa-8eed-049e9b250e2d', leaderUrl='https://redacted-url-11/', offset=967, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorSourceConnector-2, MirrorHeartbeatConnector-0], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:44,146 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-20, groupId=bdkfkbup007-mm2] Starting connectors and tasks using config offset 967 2022-08-22 17:54:44,146 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-20, groupId=bdkfkbup007-mm2] Finished starting connectors and tasks 2022-08-22 17:54:44,256 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-18, groupId=lkc-6k37mq-mm2] Attempt to heartbeat failed since group is rebalancing 2022-08-22 17:54:44,256 INFO org.apache.kafka.connect.runtime.distributed.WorkerCoordinator: [Worker clientId=connect-18, groupId=lkc-6k37mq-mm2] Rebalance started 2022-08-22 17:54:44,256 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-18, groupId=lkc-6k37mq-mm2] (Re-)joining group 2022-08-22 17:54:44,285 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-18, groupId=lkc-6k37mq-mm2] Successfully joined group with generation 2 2022-08-22 17:54:44,285 INFO org.apache.kafka.connect.runtime.Worker: Stopping connector MirrorSourceConnector 2022-08-22 17:54:44,288 INFO org.apache.kafka.connect.runtime.Worker: Stopped connector MirrorSourceConnector 2022-08-22 17:54:44,288 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-18, groupId=lkc-6k37mq-mm2] Finished stopping tasks in preparation for rebalance 2022-08-22 17:54:44,326 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-18, groupId=lkc-6k37mq-mm2] Finished flushing status backing store in preparation for rebalance 2022-08-22 17:54:44,326 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-18, groupId=lkc-6k37mq-mm2] Joined group at generation 2 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-18-4d88e151-70e5-4d76-96c9-696e1fdb87d2', leaderUrl='https://redacted-url-7/', offset=329, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorHeartbeatConnector-0], revokedConnectorIds=[MirrorSourceConnector], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:44,326 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-18, groupId=lkc-6k37mq-mm2] Starting connectors and tasks using config offset 329 2022-08-22 17:54:44,326 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-18, groupId=lkc-6k37mq-mm2] Finished starting connectors and tasks 2022-08-22 17:54:44,326 INFO org.apache.kafka.connect.runtime.distributed.WorkerCoordinator: [Worker clientId=connect-18, groupId=lkc-6k37mq-mm2] Rebalance started 2022-08-22 17:54:44,326 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-18, groupId=lkc-6k37mq-mm2] (Re-)joining group 2022-08-22 17:54:44,657 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-16, groupId=lkc-w727wj-mm2] Successfully joined group with generation 3 2022-08-22 17:54:44,658 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-16, groupId=lkc-w727wj-mm2] Joined group at generation 3 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-16-22a1e53e-3c29-42d6-9c07-6f96978db84b', leaderUrl='https://redacted-url-12/', offset=328, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorHeartbeatConnector-0], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:44,658 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-16, groupId=lkc-w727wj-mm2] Starting connectors and tasks using config offset 328 2022-08-22 17:54:44,658 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-16, groupId=lkc-w727wj-mm2] Finished starting connectors and tasks 2022-08-22 17:54:44,791 INFO org.apache.kafka.connect.mirror.Scheduler: starting checkpoint connector took 3307 ms 2022-08-22 17:54:44,792 INFO org.apache.kafka.connect.mirror.MirrorCheckpointConnector: Started MirrorCheckpointConnector with 0 consumer groups. 2022-08-22 17:54:44,792 INFO org.apache.kafka.connect.runtime.Worker: Finished creating connector MirrorCheckpointConnector 2022-08-22 17:54:44,793 INFO org.apache.kafka.connect.runtime.SourceConnectorConfig: SourceConnectorConfig values: config.action.reload = restart connector.class = org.apache.kafka.connect.mirror.MirrorCheckpointConnector errors.log.enable = false errors.log.include.messages = false errors.retry.delay.max.ms = 60000 errors.retry.timeout = 0 errors.tolerance = none header.converter = null key.converter = null name = MirrorCheckpointConnector tasks.max = 3 transforms = [] value.converter = null 2022-08-22 17:54:44,793 INFO org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig: EnrichedConnectorConfig values: config.action.reload = restart connector.class = org.apache.kafka.connect.mirror.MirrorCheckpointConnector errors.log.enable = false errors.log.include.messages = false errors.retry.delay.max.ms = 60000 errors.retry.timeout = 0 errors.tolerance = none header.converter = null key.converter = null name = MirrorCheckpointConnector tasks.max = 3 transforms = [] value.converter = null 2022-08-22 17:54:44,793 INFO org.apache.kafka.connect.runtime.distributed.WorkerCoordinator: [Worker clientId=connect-8, groupId=bdkfkwx006-mm2] Rebalance started 2022-08-22 17:54:44,793 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-8, groupId=bdkfkwx006-mm2] (Re-)joining group 2022-08-22 17:54:44,822 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-8, groupId=bdkfkwx006-mm2] Successfully joined group with generation 64 2022-08-22 17:54:44,823 INFO org.apache.kafka.connect.runtime.Worker: Stopping task MirrorSourceConnector-0 2022-08-22 17:54:44,823 INFO org.apache.kafka.connect.runtime.Worker: Stopping task MirrorSourceConnector-1 2022-08-22 17:54:44,823 INFO org.apache.kafka.connect.runtime.Worker: Stopping connector MirrorSourceConnector 2022-08-22 17:54:44,823 INFO org.apache.kafka.connect.runtime.WorkerSourceTask: WorkerSourceTask{id=MirrorSourceConnector-0} Committing offsets 2022-08-22 17:54:44,823 INFO org.apache.kafka.connect.runtime.WorkerSourceTask: WorkerSourceTask{id=MirrorSourceConnector-1} Committing offsets 2022-08-22 17:54:44,830 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=producer-65] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 2022-08-22 17:54:44,830 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=producer-64] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 2022-08-22 17:54:44,830 INFO com.cloudera.dim.mirror.ConfigConsumer: ConfigConsumer stopped. 2022-08-22 17:54:44,831 INFO org.apache.kafka.connect.mirror.MirrorSourceConnector: Stopping MirrorSourceConnector took 8 ms. 2022-08-22 17:54:44,832 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=producer-59] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 2022-08-22 17:54:44,836 INFO org.apache.kafka.connect.mirror.MirrorSourceTask: Stopping StartAndStopExecutor-connect-8-5 took 13 ms. 2022-08-22 17:54:44,836 INFO org.apache.kafka.connect.runtime.WorkerSourceTask: WorkerSourceTask{id=MirrorSourceConnector-1} flushing 0 outstanding messages for offset commit 2022-08-22 17:54:44,836 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=producer-58] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 2022-08-22 17:54:44,836 INFO org.apache.kafka.connect.runtime.Worker: Stopped connector MirrorSourceConnector 2022-08-22 17:54:44,836 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=connector-producer-MirrorSourceConnector-1] Closing the Kafka producer with timeoutMillis = 30000 ms. 2022-08-22 17:54:44,837 INFO org.apache.kafka.connect.mirror.MirrorSourceTask: Stopping StartAndStopExecutor-connect-8-7 took 14 ms. 2022-08-22 17:54:44,837 INFO org.apache.kafka.connect.runtime.WorkerSourceTask: WorkerSourceTask{id=MirrorSourceConnector-0} flushing 0 outstanding messages for offset commit 2022-08-22 17:54:44,837 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=connector-producer-MirrorSourceConnector-0] Closing the Kafka producer with timeoutMillis = 30000 ms. 2022-08-22 17:54:44,839 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-8, groupId=bdkfkwx006-mm2] Finished stopping tasks in preparation for rebalance 2022-08-22 17:54:44,842 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-8, groupId=bdkfkwx006-mm2] Finished flushing status backing store in preparation for rebalance 2022-08-22 17:54:44,842 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-8, groupId=bdkfkwx006-mm2] Joined group at generation 64 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-8-2ab025b9-cf40-40ff-973f-3ba7f2141b0c', leaderUrl='https://redacted-url-6/', offset=921, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorSourceConnector-2, MirrorHeartbeatConnector-0], revokedConnectorIds=[MirrorSourceConnector], revokedTaskIds=[MirrorSourceConnector-0, MirrorSourceConnector-1], delay=0} with rebalance delay: 0 2022-08-22 17:54:44,843 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-8, groupId=bdkfkwx006-mm2] Starting connectors and tasks using config offset 921 2022-08-22 17:54:44,843 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-8, groupId=bdkfkwx006-mm2] Finished starting connectors and tasks 2022-08-22 17:54:44,843 INFO org.apache.kafka.connect.runtime.distributed.WorkerCoordinator: [Worker clientId=connect-8, groupId=bdkfkwx006-mm2] Rebalance started 2022-08-22 17:54:44,843 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-8, groupId=bdkfkwx006-mm2] (Re-)joining group 2022-08-22 17:54:44,871 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-15, groupId=bdkfkbup007-mm2] Successfully joined group with generation 3 2022-08-22 17:54:44,871 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-15, groupId=bdkfkbup007-mm2] Joined group at generation 3 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-15-993efc86-e7dd-4ec2-b200-8f166a880349', leaderUrl='https://redacted-url-5/', offset=329, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorHeartbeatConnector-0], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:44,872 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-15, groupId=bdkfkbup007-mm2] Starting connectors and tasks using config offset 329 2022-08-22 17:54:44,872 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-15, groupId=bdkfkbup007-mm2] Finished starting connectors and tasks 2022-08-22 17:54:45,170 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-11, groupId=lkc-6k37mq-mm2] Successfully joined group with generation 3 2022-08-22 17:54:45,171 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-11, groupId=lkc-6k37mq-mm2] Joined group at generation 3 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-11-cc1d042e-23f0-475d-afc7-5302d7780736', leaderUrl='https://redacted-url-4/', offset=490, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorHeartbeatConnector-0], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:45,171 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-11, groupId=lkc-6k37mq-mm2] Starting connectors and tasks using config offset 490 2022-08-22 17:54:45,171 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-11, groupId=lkc-6k37mq-mm2] Finished starting connectors and tasks 2022-08-22 17:54:45,227 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-17, groupId=bdkfkwx006-mm2] Successfully joined group with generation 3 2022-08-22 17:54:45,227 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-17, groupId=bdkfkwx006-mm2] Joined group at generation 3 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-17-0760af01-d385-4591-93a7-0515a4830c83', leaderUrl='https://redacted-url-3/', offset=331, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorHeartbeatConnector-0], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:45,227 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-17, groupId=bdkfkwx006-mm2] Starting connectors and tasks using config offset 331 2022-08-22 17:54:45,227 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-17, groupId=bdkfkwx006-mm2] Finished starting connectors and tasks 2022-08-22 17:54:45,778 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-13, groupId=lkc-81d0q7-mm2] Successfully joined group with generation 3 2022-08-22 17:54:45,779 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-13, groupId=lkc-81d0q7-mm2] Joined group at generation 3 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-13-e3531126-99a0-4690-821d-c81975d67992', leaderUrl='https://redacted-url-2/', offset=338, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorHeartbeatConnector-0], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:45,779 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-13, groupId=lkc-81d0q7-mm2] Starting connectors and tasks using config offset 338 2022-08-22 17:54:45,779 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-13, groupId=lkc-81d0q7-mm2] Finished starting connectors and tasks 2022-08-22 17:54:46,134 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-14, groupId=bdkfkwx006-mm2] Successfully joined group with generation 3 2022-08-22 17:54:46,135 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-14, groupId=bdkfkwx006-mm2] Joined group at generation 3 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-14-db197b17-96f4-4237-ad46-723c90444e08', leaderUrl='https://redacted-url//', offset=1157, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorHeartbeatConnector-0], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:46,135 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-14, groupId=bdkfkwx006-mm2] Starting connectors and tasks using config offset 1157 2022-08-22 17:54:46,135 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-14, groupId=bdkfkwx006-mm2] Finished starting connectors and tasks 2022-08-22 17:54:46,542 ERROR org.apache.kafka.connect.runtime.Worker: Graceful stop of task MirrorSourceConnector-0 failed. 2022-08-22 17:54:46,542 ERROR org.apache.kafka.connect.runtime.Worker: Graceful stop of task MirrorSourceConnector-1 failed. 2022-08-22 17:54:46,543 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-7, groupId=kfkhwshared00-mm2] Finished stopping tasks in preparation for rebalance 2022-08-22 17:54:46,543 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-7, groupId=kfkhwshared00-mm2] Finished flushing status backing store in preparation for rebalance 2022-08-22 17:54:46,543 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-7, groupId=kfkhwshared00-mm2] Joined group at generation 2 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-7-71c64985-05d0-4c63-a59d-2b9c3a4f73c4', leaderUrl='https://redacted-url-14/', offset=507, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorSourceConnector-2, MirrorHeartbeatConnector-0], revokedConnectorIds=[MirrorSourceConnector], revokedTaskIds=[MirrorSourceConnector-0, MirrorSourceConnector-1], delay=0} with rebalance delay: 0 2022-08-22 17:54:46,543 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-7, groupId=kfkhwshared00-mm2] Starting connectors and tasks using config offset 507 2022-08-22 17:54:46,543 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-7, groupId=kfkhwshared00-mm2] Finished starting connectors and tasks 2022-08-22 17:54:46,543 ERROR org.apache.kafka.connect.storage.OffsetStorageReaderImpl: Failed to fetch offsets from namespace MirrorSourceConnector: org.apache.kafka.connect.errors.ConnectException: Offset reader closed while attempting to read offsets. This is likely because the task was been scheduled to stop but has taken longer than the graceful shutdown period to do so. at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offsets(OffsetStorageReaderImpl.java:103) at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offset(OffsetStorageReaderImpl.java:63) at org.apache.kafka.connect.mirror.MirrorSourceTask.loadOffset(MirrorSourceTask.java:234) at org.apache.kafka.connect.mirror.MirrorSourceTask.lambda$loadOffsets$4(MirrorSourceTask.java:229) at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321) at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1580) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) at org.apache.kafka.connect.mirror.MirrorSourceTask.loadOffsets(MirrorSourceTask.java:229) at org.apache.kafka.connect.mirror.MirrorSourceTask.start(MirrorSourceTask.java:99) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:213) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2022-08-22 17:54:46,543 INFO org.apache.kafka.connect.runtime.distributed.WorkerCoordinator: [Worker clientId=connect-7, groupId=kfkhwshared00-mm2] Rebalance started 2022-08-22 17:54:46,543 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-7, groupId=kfkhwshared00-mm2] (Re-)joining group 2022-08-22 17:54:46,543 ERROR org.apache.kafka.connect.storage.OffsetStorageReaderImpl: Failed to fetch offsets from namespace MirrorSourceConnector: org.apache.kafka.connect.errors.ConnectException: Offset reader closed while attempting to read offsets. This is likely because the task was been scheduled to stop but has taken longer than the graceful shutdown period to do so. at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offsets(OffsetStorageReaderImpl.java:103) at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offset(OffsetStorageReaderImpl.java:63) at org.apache.kafka.connect.mirror.MirrorSourceTask.loadOffset(MirrorSourceTask.java:234) at org.apache.kafka.connect.mirror.MirrorSourceTask.lambda$loadOffsets$4(MirrorSourceTask.java:229) at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321) at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1580) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) at org.apache.kafka.connect.mirror.MirrorSourceTask.loadOffsets(MirrorSourceTask.java:229) at org.apache.kafka.connect.mirror.MirrorSourceTask.start(MirrorSourceTask.java:99) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:213) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2022-08-22 17:54:46,544 INFO org.apache.kafka.connect.runtime.WorkerSourceTask: WorkerSourceTask{id=MirrorSourceConnector-1} Committing offsets 2022-08-22 17:54:46,544 INFO org.apache.kafka.connect.runtime.WorkerSourceTask: WorkerSourceTask{id=MirrorSourceConnector-1} flushing 0 outstanding messages for offset commit 2022-08-22 17:54:46,543 INFO org.apache.kafka.connect.runtime.WorkerSourceTask: WorkerSourceTask{id=MirrorSourceConnector-0} Committing offsets 2022-08-22 17:54:46,544 INFO org.apache.kafka.connect.runtime.WorkerSourceTask: WorkerSourceTask{id=MirrorSourceConnector-0} flushing 0 outstanding messages for offset commit 2022-08-22 17:54:46,544 ERROR org.apache.kafka.connect.runtime.WorkerTask: WorkerSourceTask{id=MirrorSourceConnector-1} Task threw an uncaught and unrecoverable exception org.apache.kafka.connect.errors.ConnectException: Failed to fetch offsets. at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offsets(OffsetStorageReaderImpl.java:114) at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offset(OffsetStorageReaderImpl.java:63) at org.apache.kafka.connect.mirror.MirrorSourceTask.loadOffset(MirrorSourceTask.java:234) at org.apache.kafka.connect.mirror.MirrorSourceTask.lambda$loadOffsets$4(MirrorSourceTask.java:229) at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321) at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1580) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) at org.apache.kafka.connect.mirror.MirrorSourceTask.loadOffsets(MirrorSourceTask.java:229) at org.apache.kafka.connect.mirror.MirrorSourceTask.start(MirrorSourceTask.java:99) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:213) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.kafka.connect.errors.ConnectException: Offset reader closed while attempting to read offsets. This is likely because the task was been scheduled to stop but has taken longer than the graceful shutdown period to do so. at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offsets(OffsetStorageReaderImpl.java:103) ... 21 common frames omitted 2022-08-22 17:54:46,544 ERROR org.apache.kafka.connect.runtime.WorkerTask: WorkerSourceTask{id=MirrorSourceConnector-0} Task threw an uncaught and unrecoverable exception org.apache.kafka.connect.errors.ConnectException: Failed to fetch offsets. at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offsets(OffsetStorageReaderImpl.java:114) at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offset(OffsetStorageReaderImpl.java:63) at org.apache.kafka.connect.mirror.MirrorSourceTask.loadOffset(MirrorSourceTask.java:234) at org.apache.kafka.connect.mirror.MirrorSourceTask.lambda$loadOffsets$4(MirrorSourceTask.java:229) at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321) at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1580) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) at org.apache.kafka.connect.mirror.MirrorSourceTask.loadOffsets(MirrorSourceTask.java:229) at org.apache.kafka.connect.mirror.MirrorSourceTask.start(MirrorSourceTask.java:99) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:213) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.kafka.connect.errors.ConnectException: Offset reader closed while attempting to read offsets. This is likely because the task was been scheduled to stop but has taken longer than the graceful shutdown period to do so. at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offsets(OffsetStorageReaderImpl.java:103) ... 21 common frames omitted 2022-08-22 17:54:46,544 ERROR org.apache.kafka.connect.runtime.WorkerTask: WorkerSourceTask{id=MirrorSourceConnector-1} Task is being killed and will not recover until manually restarted 2022-08-22 17:54:46,544 ERROR org.apache.kafka.connect.runtime.WorkerTask: WorkerSourceTask{id=MirrorSourceConnector-0} Task is being killed and will not recover until manually restarted 2022-08-22 17:54:46,545 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=producer-91] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 2022-08-22 17:54:46,545 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=producer-90] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 2022-08-22 17:54:46,547 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=producer-86] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 2022-08-22 17:54:46,547 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=producer-88] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 2022-08-22 17:54:46,549 INFO org.apache.kafka.connect.mirror.MirrorSourceTask: Stopping task-thread-MirrorSourceConnector-0 took 5 ms. 2022-08-22 17:54:46,549 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=connector-producer-MirrorSourceConnector-0] Closing the Kafka producer with timeoutMillis = 30000 ms. 2022-08-22 17:54:46,549 INFO org.apache.kafka.connect.mirror.MirrorSourceTask: Stopping task-thread-MirrorSourceConnector-1 took 5 ms. 2022-08-22 17:54:46,549 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=connector-producer-MirrorSourceConnector-1] Closing the Kafka producer with timeoutMillis = 30000 ms. 2022-08-22 17:54:47,363 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-18, groupId=lkc-6k37mq-mm2] Successfully joined group with generation 3 2022-08-22 17:54:47,364 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-18, groupId=lkc-6k37mq-mm2] Joined group at generation 3 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-18-4d88e151-70e5-4d76-96c9-696e1fdb87d2', leaderUrl='https://redacted-url-7/', offset=329, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorHeartbeatConnector-0], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:47,364 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-18, groupId=lkc-6k37mq-mm2] Starting connectors and tasks using config offset 329 2022-08-22 17:54:47,364 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-18, groupId=lkc-6k37mq-mm2] Finished starting connectors and tasks 2022-08-22 17:54:47,602 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-7, groupId=kfkhwshared00-mm2] Successfully joined group with generation 3 2022-08-22 17:54:47,603 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-7, groupId=kfkhwshared00-mm2] Joined group at generation 3 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-7-71c64985-05d0-4c63-a59d-2b9c3a4f73c4', leaderUrl='https://redacted-url-14/', offset=507, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorSourceConnector-2, MirrorHeartbeatConnector-0], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:47,603 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-7, groupId=kfkhwshared00-mm2] Starting connectors and tasks using config offset 507 2022-08-22 17:54:47,603 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-7, groupId=kfkhwshared00-mm2] Finished starting connectors and tasks 2022-08-22 17:54:47,842 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-8, groupId=bdkfkwx006-mm2] Successfully joined group with generation 65 2022-08-22 17:54:47,842 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-8, groupId=bdkfkwx006-mm2] Joined group at generation 65 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-8-2ab025b9-cf40-40ff-973f-3ba7f2141b0c', leaderUrl='https://redacted-url-6/', offset=921, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorSourceConnector-2, MirrorHeartbeatConnector-0], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:47,842 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-8, groupId=bdkfkwx006-mm2] Starting connectors and tasks using config offset 921 2022-08-22 17:54:47,842 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-8, groupId=bdkfkwx006-mm2] Finished starting connectors and tasks 2022-08-22 17:54:47,997 ERROR org.apache.kafka.connect.runtime.Worker: Graceful stop of task MirrorSourceConnector-0 failed. 2022-08-22 17:54:47,997 ERROR org.apache.kafka.connect.runtime.Worker: Graceful stop of task MirrorSourceConnector-1 failed. 2022-08-22 17:54:47,998 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-2, groupId=kfkhwshared00-mm2] Finished stopping tasks in preparation for rebalance 2022-08-22 17:54:47,998 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-2, groupId=kfkhwshared00-mm2] Finished flushing status backing store in preparation for rebalance 2022-08-22 17:54:47,998 ERROR org.apache.kafka.connect.storage.OffsetStorageReaderImpl: Failed to fetch offsets from namespace MirrorSourceConnector: org.apache.kafka.connect.errors.ConnectException: Offset reader closed while attempting to read offsets. This is likely because the task was been scheduled to stop but has taken longer than the graceful shutdown period to do so. at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offsets(OffsetStorageReaderImpl.java:103) at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offset(OffsetStorageReaderImpl.java:63) at org.apache.kafka.connect.mirror.MirrorSourceTask.loadOffset(MirrorSourceTask.java:234) at org.apache.kafka.connect.mirror.MirrorSourceTask.lambda$loadOffsets$4(MirrorSourceTask.java:229) at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321) at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1580) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) at org.apache.kafka.connect.mirror.MirrorSourceTask.loadOffsets(MirrorSourceTask.java:229) at org.apache.kafka.connect.mirror.MirrorSourceTask.start(MirrorSourceTask.java:99) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:213) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2022-08-22 17:54:47,998 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-2, groupId=kfkhwshared00-mm2] Joined group at generation 2 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-2-13dae58a-89e7-4adb-b3a3-0a30976cc7b9', leaderUrl='https://redacted-url-15/', offset=1169, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorSourceConnector-2, MirrorHeartbeatConnector-0, MirrorCheckpointConnector-0], revokedConnectorIds=[MirrorSourceConnector], revokedTaskIds=[MirrorSourceConnector-0, MirrorSourceConnector-1], delay=0} with rebalance delay: 0 2022-08-22 17:54:47,998 ERROR org.apache.kafka.connect.storage.OffsetStorageReaderImpl: Failed to fetch offsets from namespace MirrorSourceConnector: org.apache.kafka.connect.errors.ConnectException: Offset reader closed while attempting to read offsets. This is likely because the task was been scheduled to stop but has taken longer than the graceful shutdown period to do so. at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offsets(OffsetStorageReaderImpl.java:103) at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offset(OffsetStorageReaderImpl.java:63) at org.apache.kafka.connect.mirror.MirrorSourceTask.loadOffset(MirrorSourceTask.java:234) at org.apache.kafka.connect.mirror.MirrorSourceTask.lambda$loadOffsets$4(MirrorSourceTask.java:229) at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321) at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1580) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) at org.apache.kafka.connect.mirror.MirrorSourceTask.loadOffsets(MirrorSourceTask.java:229) at org.apache.kafka.connect.mirror.MirrorSourceTask.start(MirrorSourceTask.java:99) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:213) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2022-08-22 17:54:47,998 INFO org.apache.kafka.connect.runtime.WorkerSourceTask: WorkerSourceTask{id=MirrorSourceConnector-0} Committing offsets 2022-08-22 17:54:47,998 INFO org.apache.kafka.connect.runtime.WorkerSourceTask: WorkerSourceTask{id=MirrorSourceConnector-1} Committing offsets 2022-08-22 17:54:47,998 INFO org.apache.kafka.connect.runtime.WorkerSourceTask: WorkerSourceTask{id=MirrorSourceConnector-0} flushing 0 outstanding messages for offset commit 2022-08-22 17:54:47,998 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-2, groupId=kfkhwshared00-mm2] Starting connectors and tasks using config offset 1169 2022-08-22 17:54:47,998 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-2, groupId=kfkhwshared00-mm2] Finished starting connectors and tasks 2022-08-22 17:54:47,998 INFO org.apache.kafka.connect.runtime.WorkerSourceTask: WorkerSourceTask{id=MirrorSourceConnector-1} flushing 0 outstanding messages for offset commit 2022-08-22 17:54:47,998 ERROR org.apache.kafka.connect.runtime.WorkerTask: WorkerSourceTask{id=MirrorSourceConnector-0} Task threw an uncaught and unrecoverable exception org.apache.kafka.connect.errors.ConnectException: Failed to fetch offsets. at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offsets(OffsetStorageReaderImpl.java:114) at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offset(OffsetStorageReaderImpl.java:63) at org.apache.kafka.connect.mirror.MirrorSourceTask.loadOffset(MirrorSourceTask.java:234) at org.apache.kafka.connect.mirror.MirrorSourceTask.lambda$loadOffsets$4(MirrorSourceTask.java:229) at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321) at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1580) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) at org.apache.kafka.connect.mirror.MirrorSourceTask.loadOffsets(MirrorSourceTask.java:229) at org.apache.kafka.connect.mirror.MirrorSourceTask.start(MirrorSourceTask.java:99) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:213) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.kafka.connect.errors.ConnectException: Offset reader closed while attempting to read offsets. This is likely because the task was been scheduled to stop but has taken longer than the graceful shutdown period to do so. at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offsets(OffsetStorageReaderImpl.java:103) ... 21 common frames omitted 2022-08-22 17:54:47,998 INFO org.apache.kafka.connect.runtime.distributed.WorkerCoordinator: [Worker clientId=connect-2, groupId=kfkhwshared00-mm2] Rebalance started 2022-08-22 17:54:47,998 ERROR org.apache.kafka.connect.runtime.WorkerTask: WorkerSourceTask{id=MirrorSourceConnector-1} Task threw an uncaught and unrecoverable exception org.apache.kafka.connect.errors.ConnectException: Failed to fetch offsets. at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offsets(OffsetStorageReaderImpl.java:114) at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offset(OffsetStorageReaderImpl.java:63) at org.apache.kafka.connect.mirror.MirrorSourceTask.loadOffset(MirrorSourceTask.java:234) at org.apache.kafka.connect.mirror.MirrorSourceTask.lambda$loadOffsets$4(MirrorSourceTask.java:229) at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321) at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1580) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) at org.apache.kafka.connect.mirror.MirrorSourceTask.loadOffsets(MirrorSourceTask.java:229) at org.apache.kafka.connect.mirror.MirrorSourceTask.start(MirrorSourceTask.java:99) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:213) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.kafka.connect.errors.ConnectException: Offset reader closed while attempting to read offsets. This is likely because the task was been scheduled to stop but has taken longer than the graceful shutdown period to do so. at org.apache.kafka.connect.storage.OffsetStorageReaderImpl.offsets(OffsetStorageReaderImpl.java:103) ... 21 common frames omitted 2022-08-22 17:54:47,998 ERROR org.apache.kafka.connect.runtime.WorkerTask: WorkerSourceTask{id=MirrorSourceConnector-1} Task is being killed and will not recover until manually restarted 2022-08-22 17:54:47,998 ERROR org.apache.kafka.connect.runtime.WorkerTask: WorkerSourceTask{id=MirrorSourceConnector-0} Task is being killed and will not recover until manually restarted 2022-08-22 17:54:47,998 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-2, groupId=kfkhwshared00-mm2] (Re-)joining group 2022-08-22 17:54:47,999 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=producer-82] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 2022-08-22 17:54:47,999 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=producer-83] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 2022-08-22 17:54:48,001 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=producer-78] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 2022-08-22 17:54:48,002 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=producer-80] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 2022-08-22 17:54:48,002 INFO org.apache.kafka.connect.mirror.MirrorSourceTask: Stopping task-thread-MirrorSourceConnector-0 took 3 ms. 2022-08-22 17:54:48,002 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=connector-producer-MirrorSourceConnector-0] Closing the Kafka producer with timeoutMillis = 30000 ms. 2022-08-22 17:54:48,003 INFO org.apache.kafka.connect.mirror.MirrorSourceTask: Stopping task-thread-MirrorSourceConnector-1 took 4 ms. 2022-08-22 17:54:48,003 INFO org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=connector-producer-MirrorSourceConnector-1] Closing the Kafka producer with timeoutMillis = 30000 ms. 2022-08-22 17:54:49,098 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator: [Worker clientId=connect-2, groupId=kfkhwshared00-mm2] Successfully joined group with generation 3 2022-08-22 17:54:49,098 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-2, groupId=kfkhwshared00-mm2] Joined group at generation 3 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-2-13dae58a-89e7-4adb-b3a3-0a30976cc7b9', leaderUrl='https://redacted-url-15/', offset=1169, connectorIds=[MirrorHeartbeatConnector, MirrorCheckpointConnector], taskIds=[MirrorSourceConnector-2, MirrorHeartbeatConnector-0, MirrorCheckpointConnector-0], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 2022-08-22 17:54:49,098 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-2, groupId=kfkhwshared00-mm2] Starting connectors and tasks using config offset 1169 2022-08-22 17:54:49,098 INFO org.apache.kafka.connect.runtime.distributed.DistributedHerder: [Worker clientId=connect-2, groupId=kfkhwshared00-mm2] Finished starting connectors and tasks 2022-08-22 17:54:52,328 INFO org.apache.kafka.clients.consumer.KafkaConsumer: [Consumer clientId=consumer-85, groupId=null] Subscribed to partition(s): redacted-topic6-1, redacted-topic7-6, redacted-topic9-7, redacted-topic3-v1-3, redacted-topic4-1, redacted-topic5-4, redacted-topic6-7, redacted-topic4-7, redacted-topic3-v1-9, redacted-topic1-0, redacted-topic2-2, redacted-topic2-8, heartbeats-0, redacted-topic1-6, redacted-topic7-0, redacted-topic8-8, redacted-topic10-5, redacted-topic8-2, redacted-topic3-v1-0, redacted-topic7-9, redacted-topic6-4, redacted-topic5-1, redacted-topic4-4, redacted-topic9-4, redacted-topic2-5, redacted-topic5-7, redacted-topic3-v1-6, redacted-topic1-3, redacted-topic1-9, redacted-topic9-1, redacted-topic10-8, redacted-topic10-2, redacted-topic8-5, redacted-topic7-3 2022-08-22 17:54:52,328 INFO org.apache.kafka.connect.mirror.MirrorSourceTask: Starting with 0 previously uncommitted partitions. 2022-08-22 17:54:52,329 INFO org.apache.kafka.clients.consumer.KafkaConsumer: [Consumer clientId=consumer-85, groupId=null] Seeking to offset 1816903051 for partition redacted-topic6-1 2022-08-22 17:54:52,329 INFO org.apache.kafka.clients.consumer.KafkaConsumer: [Consumer clientId=consumer-85, groupId=null] Seeking to offset 220156 for partition redacted-topic7-6 2022-08-22 17:54:52,329 INFO org.apache.kafka.clients.consumer.KafkaConsumer: [Consumer clientId=consumer-85, groupId=null] Seeking to offset 2820 for partition redacted-topic9-7 2022-08-22 17:54:52,329 INFO org.apache.kafka.clients.consumer.KafkaConsumer: [Consumer clientId=consumer-85, groupId=null] Seeking to offset 101337 for partition redacted-topic3-v1-3 2022-08-22 17:54:52,329 INFO org.apache.kafka.clients.consumer.KafkaConsumer: [Consumer clientId=consumer-85, groupId=null] Seeking to offset 281816264 for partition redacted-topic4-1 2022-08-22 17:54:52,329 INFO org.apache.kafka.clients.consumer.KafkaConsumer: [Consumer clientId=consumer-85, groupId=null] Seeking to offset 33190 for partition redacted-topic5-4 2022-08-22 17:54:52,329 INFO org.apache.kafka.clients.consumer.KafkaConsumer: [Consumer clientId=consumer-85, groupId=null] Seeking to offset 1533924193 for partition redacted-topic6-7 2022-08-22 17:54:52,329 INFO org.apache.kafka.clients.consumer.KafkaConsumer: [Consumer clientId=consumer-85, groupId=null] Seeking to offset 283053940 for partition redacted-topic4-7 2022-08-22 17:54:52,329 INFO org.apache.kafka.clients.consumer.KafkaConsumer: [Consumer clientId=consumer-85, groupId=null] Seeking to offset 99506 for partition redacted-topic3-v1-9 2022-08-22 17:54:52,329 INFO org.apache.kafka.clients.consumer.KafkaConsumer: [Consumer clientId=consumer-85, groupId=null] Seeking to offset 3102688 for partition redacted-topic1-0 2022-08-22 17:54:52,329 INFO org.apache.kafka.clients.consumer.KafkaConsumer: [Consumer clientId=consumer-85, groupId=null] Seeking to offset 2145571 for partition redacted-topic2-2 2022-08-22 17:54:52,329 INFO org.apache.kafka.clients.consumer.KafkaConsumer: [Consumer clientId=consumer-85, groupId=null] Seeking to offset 2148929 for partition redacted-topic2-8 2022-08-22 17:54:52,329 INFO org.apache.kafka.clients.consumer.KafkaConsumer: [Consumer clientId=consumer-85, groupId=null] Seeking to offset 254359490 for partition heartbeats-0