NODE_1 2020-04-15 21:11:33.942 DEBUG 1 --- [-StreamThread-2] c.g.f.c.s.validation.CommandIdValidator : CommandId: mnl_cmd_9f1752da-45b7-4ef7-9ef8-209d826530bc is not a duplicate. NODE_1 2020-04-15 21:11:33.961 ERROR 1 --- [-StreamThread-2] o.a.k.s.p.i.AssignedStreamsTasks : stream-thread [XXXXCommandProcessor-dd0be3bd-1c94-4eec-b828-091f2a1a322e-StreamThread-2] Failed to process stream task 1_2 due to the following error: NODE_3 2020-04-15 21:11:47.024 INFO 1 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED NODE_3 2020-04-15 21:11:47.027 DEBUG 1 --- [-StreamThread-1] o.a.k.s.p.i.AssignedStandbyTasks : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] Closing revoked standby tasks {1_2=[mnl.xxxx.command-2, xxxx.command-2]} NODE_3 2020-04-15 21:11:47.027 DEBUG 1 --- [-StreamThread-1] o.a.k.s.processor.internals.StandbyTask : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] standby-task [1_2] Closing NODE_3 2020-04-15 21:11:47.027 TRACE 1 --- [-StreamThread-1] o.a.k.s.processor.internals.StandbyTask : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] standby-task [1_2] Committing NODE_3 2020-04-15 21:11:47.027 DEBUG 1 --- [-StreamThread-1] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] standby-task [1_2] Flushing all stores registered in the state manager NODE_3 2020-04-15 21:11:47.032 TRACE 1 --- [-StreamThread-1] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] standby-task [1_2] Flushing store COMMAND_ID_STORE NODE_3 2020-04-15 21:11:47.194 TRACE 1 --- [-StreamThread-1] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] standby-task [1_2] Flushing store XXXX_STATE_STORE NODE_3 2020-04-15 21:11:47.195 TRACE 1 --- [-StreamThread-1] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] standby-task [1_2] Checkpointable offsets updated with restored offsets: {XXXXCommandProcessor-COMMAND_ID_STORE-changelog-2=1, XXXXCommandProcessor-XXXX_STATE_STORE-changelog-2=1} NODE_3 2020-04-15 21:11:47.195 TRACE 1 --- [-StreamThread-1] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] standby-task [1_2] Checkpointable offsets updated with active acked offsets: {XXXXCommandProcessor-COMMAND_ID_STORE-changelog-2=1, XXXXCommandProcessor-XXXX_STATE_STORE-changelog-2=1} NODE_3 2020-04-15 21:11:47.195 TRACE 1 --- [-StreamThread-1] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] standby-task [1_2] Writing checkpoint: {XXXXCommandProcessor-COMMAND_ID_STORE-changelog-2=1, XXXXCommandProcessor-XXXX_STATE_STORE-changelog-2=1} NODE_3 2020-04-15 21:11:47.296 TRACE 1 --- [-StreamThread-1] o.a.k.s.processor.internals.StandbyTask : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] standby-task [1_2] Closing state manager NODE_3 2020-04-15 21:11:47.296 DEBUG 1 --- [-StreamThread-1] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] standby-task [1_2] Closing its state manager and all the registered state stores NODE_3 2020-04-15 21:11:47.298 DEBUG 1 --- [-StreamThread-1] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] standby-task [1_2] Closing storage engine COMMAND_ID_STORE NODE_3 2020-04-15 21:11:47.388 DEBUG 1 --- [-StreamThread-1] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] standby-task [1_2] Closing storage engine XXXX_STATE_STORE NODE_3 2020-04-15 21:11:47.455 DEBUG 1 --- [-StreamThread-1] o.a.k.s.p.internals.StateDirectory : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] Released state dir lock for task 1_2 NODE_3 2020-04-15 21:11:46.556 DEBUG 1 --- [-StreamThread-2] o.a.k.s.processor.internals.TaskManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] New active tasks to be created: {1_2=[mnl.xxxx.command-2, xxxx.command-2]} NODE_3 2020-04-15 21:11:46.697 TRACE 1 --- [-StreamThread-2] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] task [1_2] Checkpointable offsets read from checkpoint: {XXXXCommandProcessor-COMMAND_ID_STORE-changelog-2=1, XXXXCommandProcessor-XXXX_STATE_STORE-changelog-2=1} NODE_3 2020-04-15 21:11:46.703 DEBUG 1 --- [-StreamThread-2] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] task [1_2] Created state store manager for task 1_2 NODE_3 2020-04-15 21:11:46.950 INFO 1 --- [-StreamThread-2] o.a.k.s.p.internals.StreamThread : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] Creating producer client for task 1_2 client.id = XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-1_2-producer NODE_3 2020-04-15 21:11:47.137 INFO 1 --- [-StreamThread-2] o.a.k.clients.producer.KafkaProducer : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Instantiated a transactional producer. NODE_3 2020-04-15 21:11:47.389 INFO 1 --- [-StreamThread-2] o.a.k.clients.producer.KafkaProducer : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Overriding the default retries config to the recommended value of 2147483647 since the idempotent producer is enabled. NODE_3 2020-04-15 21:11:47.403 INFO 1 --- [-StreamThread-2] o.a.k.clients.producer.KafkaProducer : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Overriding the default acks to all since idempotence is enabled. NODE_3 2020-04-15 21:11:47.472 DEBUG 1 --- [-2-1_2-producer] o.a.k.clients.producer.internals.Sender : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Starting Kafka producer I/O thread. NODE_3 2020-04-15 21:11:47.599 INFO 1 --- [-2-1_2-producer] org.apache.kafka.clients.Metadata : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Cluster ID: UoTd5Q9HQsKwSUpY3eABQA NODE_3 2020-04-15 21:11:47.670 DEBUG 1 --- [-StreamThread-2] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] task [1_2] Register global stores [] NODE_3 2020-04-15 21:11:47.742 DEBUG 1 --- [-StreamThread-2] o.a.k.c.p.internals.TransactionManager : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Transition from state UNINITIALIZED to INITIALIZING NODE_3 2020-04-15 21:11:47.742 INFO 1 --- [-StreamThread-2] o.a.k.c.p.internals.TransactionManager : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] ProducerId set to -1 with epoch -1 NODE_3 2020-04-15 21:11:47.742 DEBUG 1 --- [-StreamThread-2] o.a.k.c.p.internals.TransactionManager : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Enqueuing transactional request InitProducerIdRequestData(transactionalId='XXXXCommandProcessor-1_2', transactionTimeoutMs=60000) NODE_3 2020-04-15 21:11:47.744 DEBUG 1 --- [-2-1_2-producer] o.a.k.c.p.internals.TransactionManager : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Enqueuing transactional request FindCoordinatorRequestData(key='XXXXCommandProcessor-1_2', keyType=1) NODE_3 2020-04-15 21:11:47.790 DEBUG 1 --- [-2-1_2-producer] o.a.k.c.p.internals.TransactionManager : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Enqueuing transactional request InitProducerIdRequestData(transactionalId='XXXXCommandProcessor-1_2', transactionTimeoutMs=60000) NODE_3 2020-04-15 21:11:47.855 DEBUG 1 --- [-2-1_2-producer] o.a.k.clients.producer.internals.Sender : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Sending transactional request FindCoordinatorRequestData(key='XXXXCommandProcessor-1_2', keyType=1) to node kafka-3:39092 (id: 3 rack: rack-a) NODE_3 2020-04-15 21:11:48.000 DEBUG 1 --- [-2-1_2-producer] o.a.k.clients.producer.internals.Sender : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Sending transactional request InitProducerIdRequestData(transactionalId='XXXXCommandProcessor-1_2', transactionTimeoutMs=60000) to node kafka-1:19092 (id: 1 rack: null) NODE_3 2020-04-15 21:11:48.133 INFO 1 --- [-2-1_2-producer] o.a.k.c.p.internals.TransactionManager : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] ProducerId set to 1014 with epoch 3 NODE_3 2020-04-15 21:11:48.146 DEBUG 1 --- [-2-1_2-producer] o.a.k.c.p.internals.TransactionManager : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Transition from state INITIALIZING to READY NODE_3 2020-04-15 21:11:48.148 TRACE 1 --- [-StreamThread-2] o.a.k.s.p.internals.StreamThread : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] Created task 1_2 with assigned partitions [mnl.xxxx.command-2, xxxx.command-2] NODE_3 2020-04-15 21:11:48.150 INFO 1 --- [-StreamThread-2] o.a.k.clients.consumer.KafkaConsumer : [Consumer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-restore-consumer, groupId=null] Unsubscribed all topics or patterns and assigned partitions NODE_3 2020-04-15 21:11:48.150 DEBUG 1 --- [-StreamThread-2] o.a.k.s.processor.internals.TaskManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] Pausing all active task partitions until the underlying state stores are ready NODE_3 2020-04-15 21:11:48.152 TRACE 1 --- [-StreamThread-2] o.a.k.s.processor.internals.TaskManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] Pausing partitions: [zzzzzz_state-1, mnl.xxxx.command-2, xxxx.command-2] NODE_3 2020-04-15 21:11:48.152 DEBUG 1 --- [-StreamThread-2] o.a.k.clients.consumer.KafkaConsumer : [Consumer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-consumer, groupId=XXXXCommandProcessor] Pausing partitions [zzzzzz_state-1, mnl.xxxx.command-2, xxxx.command-2] NODE_3 2020-04-15 21:11:48.153 INFO 1 --- [-StreamThread-2] o.a.k.s.p.internals.StreamThread : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] partition assignment took 1599 ms. NODE_3 2020-04-15 21:11:48.192 DEBUG 1 --- [-StreamThread-2] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-consumer, groupId=XXXXCommandProcessor] Fetching committed offsets for partitions: [mnl.xxxx.command-2, xxxx.command-2] NODE_3 2020-04-15 21:11:48.316 DEBUG 1 --- [-StreamThread-2] o.a.k.s.p.i.AssignedStreamsTasks : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] Initializing stream tasks [1_2] NODE_3 2020-04-15 21:11:48.326 INFO 1 --- [-StreamThread-2] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-consumer, groupId=XXXXCommandProcessor] Found no committed offset for partition mnl.xxxx.command-2 NODE_3 2020-04-15 21:11:48.365 TRACE 1 --- [-StreamThread-2] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] task [1_2] Updating store offset limit with {mnl.xxxx.command-2=0, xxxx.command-2=0} NODE_3 2020-04-15 21:11:48.371 DEBUG 1 --- [-StreamThread-2] o.a.k.s.processor.internals.StreamTask : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] task [1_2] A committed timestamp was detected: setting the partition time of partition xxxx.command-2 to 1586985068274 in stream task 1_2 NODE_3 2020-04-15 21:11:48.371 DEBUG 1 --- [-StreamThread-2] o.a.k.s.processor.internals.StreamTask : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] task [1_2] No committed offset for partition mnl.xxxx.command-2, therefore no timestamp can be found for this partition NODE_3 2020-04-15 21:11:48.393 DEBUG 1 --- [-StreamThread-2] o.a.k.s.processor.internals.StreamTask : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] task [1_2] Initializing state stores NODE_3 2020-04-15 21:11:48.410 DEBUG 1 --- [-StreamThread-2] o.a.k.s.p.internals.StateDirectory : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] Acquired state dir lock for task 1_2 NODE_3 2020-04-15 21:11:48.410 TRACE 1 --- [-StreamThread-2] o.a.k.s.processor.internals.StreamTask : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] task [1_2] Initializing state stores NODE_3 2020-04-15 21:11:48.410 DEBUG 1 --- [-StreamThread-2] o.a.k.s.processor.internals.StreamTask : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] task [1_2] Initializing store COMMAND_ID_STORE NODE_3 2020-04-15 21:11:49.075 DEBUG 1 --- [-StreamThread-2] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] task [1_2] Registering state store COMMAND_ID_STORE to its state manager NODE_3 2020-04-15 21:11:49.075 TRACE 1 --- [-StreamThread-2] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] task [1_2] Restoring state store COMMAND_ID_STORE from changelog topic XXXXCommandProcessor-COMMAND_ID_STORE-changelog at checkpoint 1 NODE_3 2020-04-15 21:11:49.076 TRACE 1 --- [-StreamThread-2] o.a.k.s.p.i.StoreChangelogReader : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] Added restorer for changelog XXXXCommandProcessor-COMMAND_ID_STORE-changelog-2 NODE_3 2020-04-15 21:11:49.076 DEBUG 1 --- [-StreamThread-2] o.a.k.s.processor.internals.StreamTask : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] task [1_2] Initializing store XXXX_STATE_STORE NODE_3 2020-04-15 21:11:49.349 DEBUG 1 --- [-StreamThread-2] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] task [1_2] Registering state store XXXX_STATE_STORE to its state manager NODE_3 2020-04-15 21:11:49.349 TRACE 1 --- [-StreamThread-2] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] task [1_2] Restoring state store XXXX_STATE_STORE from changelog topic XXXXCommandProcessor-XXXX_STATE_STORE-changelog at checkpoint 1 NODE_3 2020-04-15 21:11:49.436 TRACE 1 --- [-StreamThread-2] o.a.k.s.p.i.StoreChangelogReader : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] Found checkpoint 1 from changelog XXXXCommandProcessor-COMMAND_ID_STORE-changelog-2 for store COMMAND_ID_STORE. NODE_3 2020-04-15 21:11:50.082 TRACE 1 --- [-StreamThread-2] o.a.k.s.p.i.AssignedStreamsTasks : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] Stream task 1_2 cannot resume processing yet since some of its changelog partitions have not completed restoring: [XXXXCommandProcessor-COMMAND_ID_STORE-changelog-2, XXXXCommandProcessor-XXXX_STATE_STORE-changelog-2] NODE_3 2020-04-15 21:11:50.627 DEBUG 1 --- [-StreamThread-2] o.a.k.s.p.i.AssignedStreamsTasks : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] Transitioning stream task 1_2 to running NODE_3 2020-04-15 21:11:50.627 TRACE 1 --- [-StreamThread-2] o.a.k.s.processor.internals.StreamTask : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] task [1_2] Initializing processor nodes of the topology NODE_3 2020-04-15 21:11:51.389 DEBUG 1 --- [-StreamThread-2] o.a.k.c.p.internals.TransactionManager : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Transition from state READY to IN_TRANSACTION NODE_3 2020-04-15 21:11:51.399 DEBUG 1 --- [-StreamThread-2] o.a.k.s.p.i.AssignedStreamsTasks : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] Stream task 1_2 completed restoration as all its changelog partitions [XXXXCommandProcessor-COMMAND_ID_STORE-changelog-2, XXXXCommandProcessor-XXXX_STATE_STORE-changelog-2] have been applied to restore state NODE_3 NODE_3 2020-04-15 21:11:52.023 DEBUG 1 --- [-StreamThread-2] c.g.f.c.s.validation.CommandIdValidator : CommandId: mnl_cmd_9f1752da-45b7-4ef7-9ef8-209d826530bc is not a duplicate. NODE_3 2020-04-15 21:11:53.683 ERROR 1 --- [-StreamThread-2] o.a.k.s.p.i.AssignedStreamsTasks : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-2] Failed to process stream task 1_2 due to the following error: NODE_3 2020-04-15 21:12:04.778 DEBUG 1 --- [-StreamThread-1] o.a.k.s.processor.internals.TaskManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] New active tasks to be created: {1_2=[mnl.xxxx.command-2, xxxx.command-2]} NODE_3 2020-04-15 21:12:04.782 TRACE 1 --- [-StreamThread-1] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] task [1_2] Checkpointable offsets read from checkpoint: {XXXXCommandProcessor-COMMAND_ID_STORE-changelog-2=1, XXXXCommandProcessor-XXXX_STATE_STORE-changelog-2=1} NODE_3 2020-04-15 21:12:04.786 DEBUG 1 --- [-StreamThread-1] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] task [1_2] Created state store manager for task 1_2 NODE_3 2020-04-15 21:12:04.786 INFO 1 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] Creating producer client for task 1_2 client.id = XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1-1_2-producer NODE_3 2020-04-15 21:12:04.795 INFO 1 --- [-StreamThread-1] o.a.k.clients.producer.KafkaProducer : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Instantiated a transactional producer. NODE_3 2020-04-15 21:12:04.809 INFO 1 --- [-StreamThread-1] o.a.k.clients.producer.KafkaProducer : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Overriding the default retries config to the recommended value of 2147483647 since the idempotent producer is enabled. NODE_3 2020-04-15 21:12:04.811 INFO 1 --- [-StreamThread-1] o.a.k.clients.producer.KafkaProducer : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Overriding the default acks to all since idempotence is enabled. NODE_3 2020-04-15 21:12:04.874 DEBUG 1 --- [-1-1_2-producer] o.a.k.clients.producer.internals.Sender : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Starting Kafka producer I/O thread. NODE_3 2020-04-15 21:12:04.886 DEBUG 1 --- [-StreamThread-1] o.a.k.s.p.i.ProcessorStateManager : stream-thread [XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1] task [1_2] Register global stores [] NODE_3 2020-04-15 21:12:04.886 DEBUG 1 --- [-StreamThread-1] o.a.k.c.p.internals.TransactionManager : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Transition from state UNINITIALIZED to INITIALIZING NODE_3 2020-04-15 21:12:04.886 INFO 1 --- [-StreamThread-1] o.a.k.c.p.internals.TransactionManager : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] ProducerId set to -1 with epoch -1 NODE_3 2020-04-15 21:12:04.886 DEBUG 1 --- [-StreamThread-1] o.a.k.c.p.internals.TransactionManager : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Enqueuing transactional request InitProducerIdRequestData(transactionalId='XXXXCommandProcessor-1_2', transactionTim NODE_3 eoutMs=60000) NODE_3 2020-04-15 21:12:04.890 DEBUG 1 --- [-1-1_2-producer] o.a.k.c.p.internals.TransactionManager : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Enqueuing transactional request FindCoordinatorRequestData(key='XXXXCommandProcessor-1_2', keyType=1) NODE_3 2020-04-15 21:12:04.890 DEBUG 1 --- [-1-1_2-producer] o.a.k.c.p.internals.TransactionManager : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Enqueuing transactional request InitProducerIdRequestData(transactionalId='XXXXCommandProcessor-1_2', transactionTim NODE_3 eoutMs=60000) NODE_3 2020-04-15 21:12:04.891 DEBUG 1 --- [-1-1_2-producer] o.a.k.clients.producer.internals.Sender : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Sending transactional request FindCoordinatorRequestData(key='XXXXCommandProcessor-1_2', keyType=1) to node kafka-3: NODE_3 39092 (id: -3 rack: null) NODE_3 2020-04-15 21:12:04.892 INFO 1 --- [-1-1_2-producer] org.apache.kafka.clients.Metadata : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Cluster ID: UoTd5Q9HQsKwSUpY3eABQA NODE_3 2020-04-15 21:12:05.002 DEBUG 1 --- [-1-1_2-producer] o.a.k.clients.producer.internals.Sender : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Sending transactional request InitProducerIdRequestData(transactionalId='XXXXCommandProcessor-1_2', transactionTimeo NODE_3 utMs=60000) to node kafka-1:19092 (id: 1 rack: null) NODE_3 2020-04-15 21:12:05.020 INFO 1 --- [-1-1_2-producer] o.a.k.c.p.internals.TransactionManager : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] ProducerId set to 1014 with epoch 4 NODE_3 2020-04-15 21:12:05.027 DEBUG 1 --- [-1-1_2-producer] o.a.k.c.p.internals.TransactionManager : [Producer clientId=XXXXCommandProcessor-94f7be8e-beec-411f-b4ec-9031527bccdf-StreamThread-1-1_2-producer, transactionalId=XXXXCommandProcessor-1_2] Transition from state INITIALIZING to READY NODE_3 2020-04-15 21:12:06.424 WARN 1 --- [-StreamThread-1] c.g.f.c.s.validation.CommandIdValidator : Command duplicate detected. Command id mnl_cmd_9f1752da-45b7-4ef7-9ef8-209d826530bc