Aug 07 10:36:12.456 | 2020-08-07 10:36:12.456 INFO 21 --- [pool-7-thread-1] e.p.EventPublishingPipelineStateListener : Sending out pipeline state change event from REBALANCING to RUNNING for pipeline: profile-transfers-summary with host i-main02-fra-production Aug 07 10:36:12.456 | stream-client [cluster1-profile-transfers-summary-client-id] State transition from REBALANCING to RUNNING Aug 07 10:36:12.456 | stream-thread [cluster1-profile-transfers-summary-client-id-StreamThread-1] standby-task [1_14] State store did not find checkpoint offset, hence would default to the starting offset at changelog cluster1-profile-transfers-summary--changelog-14 Aug 07 10:36:12.456 | 2020-08-07 10:36:12.455 INFO 21 --- [-StreamThread-1] o.a.k.s.p.i.ProcessorStateManager : stream-thread [cluster1-profile-transfers-summary-client-id-StreamThread-1] standby-task [1_14] State store did not find checkpoint offset, hence would default to the starting offset at changelog cluster1-profile-transfers-summary--changelog-14 Aug 07 10:36:12.456 | stream-thread [cluster1-profile-transfers-summary-client-id-StreamThread-1] standby-task [1_14] Initialized Aug 07 10:36:12.456 | 2020-08-07 10:36:12.456 INFO 21 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread : stream-thread [cluster1-profile-transfers-summary-client-id-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING Aug 07 10:36:12.457 | [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Subscribed to partition(s): cluster1-profile-transfers-summary-northam_total_summary_events_store-changelog-8, cluster1-profile-transfers-summary-profile-transfers-summary-store-changelog-7, cluster1-profile-transfers-summary--changelog-2, cluster1-profile-transfers-summary--changelog-14, cluster1-profile-transfers-summary--changelog-15, cluster1-profile-transfers-summary-northam_total_summary_events_store-changelog-2, cluster1-profile-transfers-summary--changelog-8, cluster1-profile-transfers-summary-northam_total_summary_events_store-changelog-14, cluster1-profile-transfers-summary-northam_total_summary_events_store-changelog-15, cluster1-profile-transfers-summary-profile-transfers-summary-store-changelog-1 Aug 07 10:36:12.457 | Sending out pipeline state change event from REBALANCING to RUNNING for pipeline: profile-transfers-summary with host i-main02-fra-production Aug 07 10:36:12.457 | 2020-08-07 10:36:12.456 INFO 21 --- [-StreamThread-1] o.a.k.clients.consumer.KafkaConsumer : [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Subscribed to partition(s): cluster1-profile-transfers-summary-northam_total_summary_events_store-changelog-8, cluster1-profile-transfers-summary-profile-transfers-summary-store-changelog-7, cluster1-profile-transfers-summary--changelog-2, cluster1-profile-transfers-summary--changelog-14, cluster1-profile-transfers-summary--changelog-15, cluster1-profile-transfers-summary-northam_total_summary_events_store-changelog-2, cluster1-profile-transfers-summary--changelog-8, cluster1-profile-transfers-summary-northam_total_summary_events_store-changelog-14, cluster1-profile-transfers-summary-northam_total_summary_events_store-changelog-15, cluster1-profile-transfers-summary-profile-transfers-summary-store-changelog-1 Aug 07 10:36:12.457 | 2020-08-07 10:36:12.457 INFO 21 --- [-StreamThread-1] o.a.k.c.c.internals.SubscriptionState : [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Seeking to EARLIEST offset of partition cluster1-profile-transfers-summary--changelog-14 Aug 07 10:36:12.457 | [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Seeking to EARLIEST offset of partition cluster1-profile-transfers-summary--changelog-14 Aug 07 10:36:12.458 | 2020-08-07 10:36:12.458 INFO 21 --- [-StreamThread-1] o.a.k.c.c.internals.SubscriptionState : [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Seeking to EARLIEST offset of partition cluster1-profile-transfers-summary-northam_total_summary_events_store-changelog-14 Aug 07 10:36:12.458 | [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Seeking to EARLIEST offset of partition cluster1-profile-transfers-summary-northam_total_summary_events_store-changelog-14 Aug 07 10:36:12.459 | 2020-08-07 10:36:12.459 INFO 21 --- [internal-client] e.p.EventPublishingPipelineStateListener : successfully sent pipeline state change event with key: cluster1-profile-transfers-summa-main02-fra-production Aug 07 10:36:12.459 | successfully sent pipeline state change event with key: cluster1-profile-transfers-summa-main02-fra-production Aug 07 10:36:12.464 | [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Failed authentication with kafka-broker1 (SSL handshake failed) Aug 07 10:36:12.464 | 2020-08-07 10:36:12.464 INFO 21 --- [-StreamThread-1] o.apache.kafka.common.network.Selector : [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Failed authentication with kafka-broker1 (SSL handshake failed) Aug 07 10:36:12.465 | [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Connection to node 1012 (kafka-broker2:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. Aug 07 10:36:12.465 | 2020-08-07 10:36:12.465 WARN 21 --- [-StreamThread-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Connection to node 1012 (kafka-broker2:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. Aug 07 10:36:12.465 | [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Connection to node 1013 (/kafka-broker4:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. Aug 07 10:36:12.465 | 2020-08-07 10:36:12.464 WARN 21 --- [-StreamThread-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Connection to node 1003 (kafka-broker3:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. Aug 07 10:36:12.465 | [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Connection to node 1003 (kafka-broker3:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. Aug 07 10:36:12.465 | 2020-08-07 10:36:12.465 WARN 21 --- [-StreamThread-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Connection to node 1013 (/kafka-broker4:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. Aug 07 10:36:12.465 | 2020-08-07 10:36:12.465 ERROR 21 --- [-StreamThread-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Connection to node 1011 (kafka-broker1:9093) failed authentication due to: SSL handshake failed Aug 07 10:36:12.470 | [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Connection to node 1010 (kafka-broker6:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. Aug 07 10:36:12.470 | 2020-08-07 10:36:12.470 WARN 21 --- [-StreamThread-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Connection to node 1010 (kafka-broker6:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. Aug 07 10:36:12.470 | [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Connection to node 1011 (kafka-broker1:9093) failed authentication due to: SSL handshake failed Aug 07 10:36:12.471 | 2020-08-07 10:36:12.471 INFO 21 --- [-StreamThread-1] o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 1013: Aug 07 10:36:12.471 | Aug 07 10:36:12.471 | org.apache.kafka.common.errors.DisconnectException: null Aug 07 10:36:12.471 | org.apache.kafka.common.errors.DisconnectException: null Aug 07 10:36:12.471 | Aug 07 10:36:12.471 | 2020-08-07 10:36:12.471 INFO 21 --- [-StreamThread-1] o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 1012: Aug 07 10:36:12.471 | Aug 07 10:36:12.471 | [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 1013: Aug 07 10:36:12.471 | Aug 07 10:36:12.471 | 2020-08-07 10:36:12.470 INFO 21 --- [-StreamThread-1] o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 1003: Aug 07 10:36:12.471 | [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 1003: Aug 07 10:36:12.471 | org.apache.kafka.common.errors.DisconnectException: null Aug 07 10:36:12.471 | Aug 07 10:36:12.471 | [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 1012: Aug 07 10:36:12.471 | Aug 07 10:36:12.472 | at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:445) Aug 07 10:36:12.472 | 2020-08-07 10:36:12.472 INFO 21 --- [-StreamThread-1] o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 1010: Aug 07 10:36:12.472 | at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1247) Aug 07 10:36:12.472 | at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:645) Aug 07 10:36:12.472 | Aug 07 10:36:12.472 | at org.apache.kafka.common.network.Selector.poll(Selector.java:485) Aug 07 10:36:12.472 | at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1237) Aug 07 10:36:12.472 | at java.base/sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144) Aug 07 10:36:12.472 | org.apache.kafka.common.errors.DisconnectException: null Aug 07 10:36:12.472 | at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) Aug 07 10:36:12.472 | at java.base/java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) Aug 07 10:36:12.472 | ... 29 common frames omitted Aug 07 10:36:12.472 | at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) Aug 07 10:36:12.472 | at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:145) Aug 07 10:36:12.472 | Caused by: java.security.cert.CertificateExpiredException: NotAfter: Fri Aug 07 07:45:16 GMT 2020 Aug 07 10:36:12.472 | at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368) Aug 07 10:36:12.472 | Aug 07 10:36:12.472 | at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:623) Aug 07 10:36:12.472 | [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 1011: Aug 07 10:36:12.472 | Caused by: java.security.cert.CertificateException: java.security.cert.CertPathValidatorException: validity check failed Aug 07 10:36:12.472 | Caused by: java.security.cert.CertPathValidatorException: validity check failed Aug 07 10:36:12.472 | Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertPathValidatorException: validity check failed Aug 07 10:36:12.472 | org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Aug 07 10:36:12.472 | at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1192) Aug 07 10:36:12.472 | at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:268) Aug 07 10:36:12.472 | at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) Aug 07 10:36:12.472 | at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) Aug 07 10:36:12.472 | at java.base/sun.security.x509.CertificateValidity.valid(CertificateValidity.java:274) Aug 07 10:36:12.472 | at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263) Aug 07 10:36:12.472 | at java.base/sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:669) Aug 07 10:36:12.472 | at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) Aug 07 10:36:12.472 | [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 1010: Aug 07 10:36:12.472 | at java.base/java.security.AccessController.doPrivileged(AccessController.java:691) Aug 07 10:36:12.472 | at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:464) Aug 07 10:36:12.472 | at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:237) Aug 07 10:36:12.472 | Aug 07 10:36:12.472 | at spiffe.provider.SpiffeTrustManager.checkPeer(SpiffeTrustManager.java:90) Aug 07 10:36:12.472 | at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514) Aug 07 10:36:12.472 | at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:669) Aug 07 10:36:12.472 | at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547) Aug 07 10:36:12.472 | at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) Aug 07 10:36:12.472 | at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:544) Aug 07 10:36:12.472 | at spiffe.provider.SpiffeTrustManager.checkServerTrusted(SpiffeTrustManager.java:49) Aug 07 10:36:12.472 | ... 25 common frames omitted Aug 07 10:36:12.472 | ... 34 common frames omitted Aug 07 10:36:12.472 | at spiffe.provider.CertificateUtils.validate(CertificateUtils.java:85) Aug 07 10:36:12.472 | at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) Aug 07 10:36:12.472 | 2020-08-07 10:36:12.472 INFO 21 --- [-StreamThread-1] o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=cluster1-profile-transfers-summary-client-id-StreamThread-1-restore-consumer, groupId=null] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 1011: Aug 07 10:36:12.472 | at spiffe.provider.SpiffeTrustManager.checkServerTrusted(SpiffeTrustManager.java:79) Aug 07 10:36:12.472 | at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) Aug 07 10:36:12.472 | at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1301) Aug 07 10:36:12.472 | at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135) Aug 07 10:36:12.472 | at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:325) Aug 07 10:36:12.472 | at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430) Aug 07 10:36:12.472 | at java.base/sun.security.provider.certpath.BasicChecker.verifyValidity(BasicChecker.java:190) Aug 07 10:36:12.472 | at java.base/sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:84) Aug 07 10:36:12.472 | Aug 07 10:36:12.472 | at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:360) Aug 07 10:36:12.472 | at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) Aug 07 10:36:12.472 | at spiffe.provider.CertificateUtils.validate(CertificateUtils.java:87) Aug 07 10:36:12.472 | at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) Aug 07 10:36:12.472 | at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173) Aug 07 10:36:12.472 | at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:433) Aug 07 10:36:12.478 | at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) Aug 07 10:36:12.478 | Aug 07 10:36:12.478 | 2020-08-07 10:36:12.478 ERROR 21 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread : stream-thread [cluster1-profile-transfers-summary-client-id-StreamThread-1] Encountered the following exception during processing and the thread is going to shut down: Aug 07 10:36:12.478 | at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:445) Aug 07 10:36:12.478 | at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:237) Aug 07 10:36:12.478 | at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) Aug 07 10:36:12.478 | at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:623) Aug 07 10:36:12.478 | at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135) Aug 07 10:36:12.478 | at spiffe.provider.SpiffeTrustManager.checkServerTrusted(SpiffeTrustManager.java:79) Aug 07 10:36:12.478 | at java.base/sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:669) Aug 07 10:36:12.478 | at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:464) Aug 07 10:36:12.478 | at java.base/sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144) Aug 07 10:36:12.478 | Aug 07 10:36:12.478 | Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Aug 07 10:36:12.478 | at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:669) Aug 07 10:36:12.478 | at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1192) Aug 07 10:36:12.478 | at org.apache.kafka.common.network.Selector.poll(Selector.java:485) Aug 07 10:36:12.478 | at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:448) Aug 07 10:36:12.478 | ... 34 common frames omitted Aug 07 10:36:12.478 | at java.base/java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) Aug 07 10:36:12.478 | at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:360) Aug 07 10:36:12.478 | at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) Aug 07 10:36:12.478 | ... 25 common frames omitted Aug 07 10:36:12.478 | at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:544) Aug 07 10:36:12.478 | at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430) Aug 07 10:36:12.478 | at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514) Aug 07 10:36:12.478 | at java.base/java.security.AccessController.doPrivileged(AccessController.java:691) Aug 07 10:36:12.478 | Caused by: java.security.cert.CertificateExpiredException: NotAfter: Fri Aug 07 07:45:16 GMT 2020 Aug 07 10:36:12.478 | at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) Aug 07 10:36:12.478 | at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) Aug 07 10:36:12.478 | at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) Aug 07 10:36:12.478 | at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:268) Aug 07 10:36:12.478 | at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) Aug 07 10:36:12.478 | at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) Aug 07 10:36:12.478 | at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1247) Aug 07 10:36:12.478 | at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1301) Aug 07 10:36:12.478 | at spiffe.provider.CertificateUtils.validate(CertificateUtils.java:85) Aug 07 10:36:12.478 | at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1237) Aug 07 10:36:12.478 | Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertPathValidatorException: validity check failed Aug 07 10:36:12.478 | at java.base/sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:84) Aug 07 10:36:12.478 | Caused by: java.security.cert.CertificateException: java.security.cert.CertPathValidatorException: validity check failed Aug 07 10:36:12.478 | at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263) Aug 07 10:36:12.478 | org.apache.kafka.streams.errors.StreamsException: Restore consumer get unexpected error polling records. Aug 07 10:36:12.478 | at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:645) Aug 07 10:36:12.478 | at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368) Aug 07 10:36:12.478 | at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:145) Aug 07 10:36:12.478 | at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:325) Aug 07 10:36:12.478 | at java.base/sun.security.x509.CertificateValidity.valid(CertificateValidity.java:274) Aug 07 10:36:12.478 | at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) Aug 07 10:36:12.478 | at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) Aug 07 10:36:12.478 | at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) Aug 07 10:36:12.478 | at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) Aug 07 10:36:12.478 | at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:433) Aug 07 10:36:12.478 | at spiffe.provider.SpiffeTrustManager.checkPeer(SpiffeTrustManager.java:90) Aug 07 10:36:12.478 | Caused by: java.security.cert.CertPathValidatorException: validity check failed Aug 07 10:36:12.478 | at spiffe.provider.CertificateUtils.validate(CertificateUtils.java:87) Aug 07 10:36:12.478 | at spiffe.provider.SpiffeTrustManager.checkServerTrusted(SpiffeTrustManager.java:49) Aug 07 10:36:12.478 | at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547) Aug 07 10:36:12.478 | ... 29 common frames omitted Aug 07 10:36:12.478 | at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173) Aug 07 10:36:12.478 | at java.base/sun.security.provider.certpath.BasicChecker.verifyValidity(BasicChecker.java:190) Aug 07 10:36:12.478 | at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:669) Aug 07 10:36:12.483 | stream-thread [cluster1-profile-transfers-summary-client-id-StreamThread-1] Encountered the following exception during processing and the thread is going to shut down: Aug 07 10:36:12.483 | 2020-08-07 10:36:12.483 INFO 21 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread : stream-thread [cluster1-profile-transfers-summary-client-id-StreamThread-1] Shutting down Aug 07 10:36:12.483 | 2020-08-07 10:36:12.483 INFO 21 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread : stream-thread [cluster1-profile-transfers-summary-client-id-StreamThread-1] State transition from RUNNING to PENDING_SHUTDOWN Aug 07 10:36:12.483 | stream-thread [cluster1-profile-transfers-summary-client-id-StreamThread-1] State transition from RUNNING to PENDING_SHUTDOWN