log4j:WARN No such property [datePattern] in org.apache.log4j.FileAppender. [2019-03-05 09:03:03,945] INFO ProducerConfig values: acks = 1 batch.size = 16384 bootstrap.servers = [mwkafka-prod-02.nyc.xxx.com:9092] buffer.memory = 33554432 client.dns.lookup = default client.id = jira_prod compression.type = none connections.max.idle.ms = 540000 delivery.timeout.ms = 120000 enable.idempotence = false interceptor.classes = [] key.serializer = class org.apache.kafka.common.serialization.StringSerializer linger.ms = 0 max.block.ms = 60000 max.in.flight.requests.per.connection = 5 max.request.size = 1048576 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner receive.buffer.bytes = 32768 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 30000 retries = 2147483647 retry.backoff.ms = 100 sasl.client.callback.handler.class = null sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = kafka sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.mechanism = GSSAPI security.protocol = SASL_PLAINTEXT send.buffer.bytes = 131072 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm = https ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS transaction.timeout.ms = 60000 transactional.id = null value.serializer = class org.apache.kafka.common.serialization.StringSerializer (org.apache.kafka.clients.producer.ProducerConfig) [2019-03-05 09:03:03,958] DEBUG Added sensor with name bufferpool-wait-time (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:03,962] DEBUG Added sensor with name buffer-exhausted-records (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:03,967] DEBUG Updated cluster metadata version 1 to Cluster(id = null, nodes = [mwkafka-prod-02.nyc.xxx.com:9092 (id: -1 rack: null)], partitions = [], controller = null) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:03,967] DEBUG Added sensor with name errors (org.apache.kafka.common.metrics.Metrics) WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.kafka.common.network.SaslChannelBuilder (file:/proj/tools/infra/apache/kafka/kafka_2.12-2.1.1/libs/kafka-clients-2.1.1.jar) to method sun.security.krb5.Config.getInstance() WARNING: Please consider reporting this to the maintainers of org.apache.kafka.common.network.SaslChannelBuilder WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Debug is true storeKey false useTicketCache true useKeyTab false doNotPrompt false ticketCache is null isInitiator true KeyTab is null refreshKrb5Config is false principal is null tryFirstPass is false useFirstPass is false storePass is false clearPass is false Acquire TGT from Cache Principal is choudhab/hostbased@UNIX.XXX.COM Commit Succeeded [2019-03-05 09:03:04,171] INFO Successfully logged in. (org.apache.kafka.common.security.authenticator.AbstractLogin) [2019-03-05 09:03:04,171] DEBUG [Principal=null]: It is a Kerberos ticket (org.apache.kafka.common.security.kerberos.KerberosLogin) [2019-03-05 09:03:04,172] INFO [Principal=null]: TGT refresh thread started. (org.apache.kafka.common.security.kerberos.KerberosLogin) [2019-03-05 09:03:04,173] DEBUG Found TGT with client principal 'choudhab/hostbased@UNIX.XXX.COM' and server principal 'krbtgt/UNIX.XXX.COM@UNIX.XXX.COM'. (org.apache.kafka.common.security.kerberos.KerberosLogin) [2019-03-05 09:03:04,173] INFO [Principal=null]: TGT valid starting at: Tue Mar 05 06:00:21 EST 2019 (org.apache.kafka.common.security.kerberos.KerberosLogin) [2019-03-05 09:03:04,173] INFO [Principal=null]: TGT expires: Tue Mar 05 14:00:21 EST 2019 (org.apache.kafka.common.security.kerberos.KerberosLogin) [2019-03-05 09:03:04,173] INFO [Principal=null]: TGT refresh sleeping until: Tue Mar 05 12:43:09 EST 2019 (org.apache.kafka.common.security.kerberos.KerberosLogin) [2019-03-05 09:03:04,176] DEBUG Added sensor with name produce-throttle-time (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,183] DEBUG Added sensor with name connections-closed: (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,183] DEBUG Added sensor with name connections-created: (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,183] DEBUG Added sensor with name successful-authentication: (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,184] DEBUG Added sensor with name failed-authentication: (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,184] DEBUG Added sensor with name bytes-sent-received: (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,185] DEBUG Added sensor with name bytes-sent: (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,185] DEBUG Added sensor with name bytes-received: (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,186] DEBUG Added sensor with name select-time: (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,187] DEBUG Added sensor with name io-time: (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,190] DEBUG Added sensor with name batch-size (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,190] DEBUG Added sensor with name compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,190] DEBUG Added sensor with name queue-time (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,191] DEBUG Added sensor with name request-time (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,191] DEBUG Added sensor with name records-per-request (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,191] DEBUG Added sensor with name record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,192] DEBUG Added sensor with name record-size (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,193] DEBUG Added sensor with name batch-split-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,193] DEBUG [Producer clientId=jira_prod] Starting Kafka producer I/O thread. (org.apache.kafka.clients.producer.internals.Sender) [2019-03-05 09:03:04,195] INFO Kafka version : 2.1.1 (org.apache.kafka.common.utils.AppInfoParser) [2019-03-05 09:03:04,195] INFO Kafka commitId : 21234bee31165527 (org.apache.kafka.common.utils.AppInfoParser) [2019-03-05 09:03:04,197] DEBUG [Producer clientId=jira_prod] Kafka producer started (org.apache.kafka.clients.producer.KafkaProducer) [2019-03-05 09:03:04,203] DEBUG [Producer clientId=jira_prod] Initialize connection to node mwkafka-prod-02.nyc.xxx.com:9092 (id: -1 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,204] DEBUG [Producer clientId=jira_prod] Initiating connection to node mwkafka-prod-02.nyc.xxx.com:9092 (id: -1 rack: null) using address mwkafka-prod-02.nyc.xxx.com/10.219.247.26 (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,211] DEBUG Set SASL client state to SEND_APIVERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,212] DEBUG Creating SaslClient: client=choudhab/hostbased@UNIX.XXX.COM;service=kafka;serviceHostname=mwkafka-prod-02.nyc.xxx.com;mechs=[GSSAPI] (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,245] DEBUG Added sensor with name node--1.bytes-sent (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,245] DEBUG Added sensor with name node--1.bytes-received (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,246] DEBUG Added sensor with name node--1.latency (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,246] DEBUG [Producer clientId=jira_prod] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1 (org.apache.kafka.common.network.Selector) [2019-03-05 09:03:04,315] DEBUG Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,315] DEBUG [Producer clientId=jira_prod] Completed connection to node -1. Fetching API versions. (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,318] DEBUG Set SASL client state to SEND_HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,319] DEBUG Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,320] DEBUG Set SASL client state to INITIAL (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,332] DEBUG Set SASL client state to INTERMEDIATE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,334] DEBUG Set SASL client state to CLIENT_COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,335] DEBUG Set SASL client state to COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,335] DEBUG [Producer clientId=jira_prod] Initiating API versions fetch from node -1. (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,339] DEBUG [Producer clientId=jira_prod] Recorded API versions for node -1: (Produce(0): 0 to 7 [usable: 7], Fetch(1): 0 to 10 [usable: 10], ListOffsets(2): 0 to 5 [usable: 4], Metadata(3): 0 to 7 [usable: 7], LeaderAndIsr(4): 0 to 2 [usable: 1], StopReplica(5): 0 to 1 [usable: 0], UpdateMetadata(6): 0 to 5 [usable: 4], ControlledShutdown(7): 0 to 2 [usable: 1], OffsetCommit(8): 0 to 6 [usable: 6], OffsetFetch(9): 0 to 5 [usable: 5], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0 to 4 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2 [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2 [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3 [usable: 3], DeleteTopics(20): 0 to 3 [usable: 3], DeleteRecords(21): 0 to 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1], OffsetForLeaderEpoch(23): 0 to 2 [usable: 2], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 2 [usable: 2], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1 [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1 [usable: 1], SaslAuthenticate(36): 0 to 1 [usable: 0], CreatePartitions(37): 0 to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1], RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0 to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1], DeleteGroups(42): 0 to 1 [usable: 1], UNKNOWN(43): 0) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,339] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-0) to node mwkafka-prod-02.nyc.xxx.com:9092 (id: -1 rack: null) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,343] INFO Cluster ID: bIfwPQCgTry9_1SkT9W-1A (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,344] DEBUG Updated cluster metadata version 2 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com), mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com), mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com), mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com), mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com)], partitions = [Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,381] DEBUG [Producer clientId=jira_prod] Initiating connection to node mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com) using address mwkafka-prod-02.tbd.xxx.com/10.236.30.31 (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,382] DEBUG Set SASL client state to SEND_APIVERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,382] DEBUG Creating SaslClient: client=choudhab/hostbased@UNIX.XXX.COM;service=kafka;serviceHostname=mwkafka-prod-02.tbd.xxx.com;mechs=[GSSAPI] (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,383] DEBUG Added sensor with name node-6.bytes-sent (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,384] DEBUG Added sensor with name node-6.bytes-received (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,384] DEBUG Added sensor with name node-6.latency (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,384] DEBUG [Producer clientId=jira_prod] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 6 (org.apache.kafka.common.network.Selector) [2019-03-05 09:03:04,384] DEBUG Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,384] DEBUG [Producer clientId=jira_prod] Completed connection to node 6. Fetching API versions. (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,385] DEBUG [Producer clientId=jira_prod] Initialize connection to node mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com) for sending metadata request (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,385] DEBUG [Producer clientId=jira_prod] Initiating connection to node mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com) using address mwkafka-prod-02.nyc.xxx.com/10.219.247.26 (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,385] DEBUG Set SASL client state to SEND_APIVERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,385] DEBUG Creating SaslClient: client=choudhab/hostbased@UNIX.XXX.COM;service=kafka;serviceHostname=mwkafka-prod-02.nyc.xxx.com;mechs=[GSSAPI] (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,386] DEBUG Set SASL client state to SEND_HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,386] DEBUG Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,386] DEBUG Added sensor with name node-2.bytes-sent (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,387] DEBUG Added sensor with name node-2.bytes-received (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,387] DEBUG Added sensor with name node-2.latency (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,388] DEBUG [Producer clientId=jira_prod] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 2 (org.apache.kafka.common.network.Selector) [2019-03-05 09:03:04,388] DEBUG Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,388] DEBUG [Producer clientId=jira_prod] Completed connection to node 2. Fetching API versions. (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,388] DEBUG [Producer clientId=jira_prod] Initialize connection to node mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com) for sending metadata request (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,390] DEBUG [Producer clientId=jira_prod] Initiating connection to node mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com) using address mwkafka-prod-01.tbd.xxx.com/10.236.30.30 (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,390] DEBUG Set SASL client state to SEND_APIVERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,390] DEBUG Creating SaslClient: client=choudhab/hostbased@UNIX.XXX.COM;service=kafka;serviceHostname=mwkafka-prod-01.tbd.xxx.com;mechs=[GSSAPI] (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,391] DEBUG Set SASL client state to SEND_HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,391] DEBUG Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,391] DEBUG Added sensor with name node-5.bytes-sent (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,392] DEBUG Added sensor with name node-5.bytes-received (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,392] DEBUG Added sensor with name node-5.latency (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,392] DEBUG [Producer clientId=jira_prod] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 5 (org.apache.kafka.common.network.Selector) [2019-03-05 09:03:04,392] DEBUG Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,392] DEBUG Set SASL client state to INITIAL (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,393] DEBUG Set SASL client state to INTERMEDIATE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,393] DEBUG [Producer clientId=jira_prod] Completed connection to node 5. Fetching API versions. (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,393] DEBUG [Producer clientId=jira_prod] Initialize connection to node mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com) for sending metadata request (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,396] DEBUG [Producer clientId=jira_prod] Initiating connection to node mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com) using address mwkafka-prod-01.dr.xxx.com/10.218.247.22 (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,396] DEBUG Set SASL client state to SEND_APIVERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,396] DEBUG Creating SaslClient: client=choudhab/hostbased@UNIX.XXX.COM;service=kafka;serviceHostname=mwkafka-prod-01.dr.xxx.com;mechs=[GSSAPI] (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,396] DEBUG Set SASL client state to INITIAL (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,397] DEBUG Set SASL client state to INTERMEDIATE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,397] DEBUG Added sensor with name node-3.bytes-sent (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,398] DEBUG Added sensor with name node-3.bytes-received (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,398] DEBUG Added sensor with name node-3.latency (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,399] DEBUG [Producer clientId=jira_prod] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 3 (org.apache.kafka.common.network.Selector) [2019-03-05 09:03:04,399] DEBUG Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,399] DEBUG Set SASL client state to SEND_HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,399] DEBUG Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,399] DEBUG Set SASL client state to CLIENT_COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,399] DEBUG [Producer clientId=jira_prod] Completed connection to node 3. Fetching API versions. (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,400] DEBUG Set SASL client state to CLIENT_COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,400] DEBUG Set SASL client state to SEND_HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,400] DEBUG Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,400] DEBUG Set SASL client state to INITIAL (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,401] DEBUG Set SASL client state to INTERMEDIATE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,401] DEBUG Set SASL client state to INITIAL (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,402] DEBUG Set SASL client state to INTERMEDIATE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,402] DEBUG Set SASL client state to COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,402] DEBUG [Producer clientId=jira_prod] Initiating API versions fetch from node 6. (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,402] DEBUG [Producer clientId=jira_prod] Initialize connection to node mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com) for sending metadata request (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,404] DEBUG [Producer clientId=jira_prod] Initiating connection to node mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com) using address mwkafka-prod-02.dr.xxx.com/10.218.247.23 (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,404] DEBUG Set SASL client state to SEND_APIVERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,404] DEBUG Creating SaslClient: client=choudhab/hostbased@UNIX.XXX.COM;service=kafka;serviceHostname=mwkafka-prod-02.dr.xxx.com;mechs=[GSSAPI] (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,405] DEBUG Set SASL client state to COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,405] DEBUG Set SASL client state to CLIENT_COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,405] DEBUG Set SASL client state to CLIENT_COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,405] DEBUG Added sensor with name node-4.bytes-sent (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,406] DEBUG Added sensor with name node-4.bytes-received (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,406] DEBUG Added sensor with name node-4.latency (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,406] DEBUG [Producer clientId=jira_prod] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 4 (org.apache.kafka.common.network.Selector) [2019-03-05 09:03:04,407] DEBUG Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,407] DEBUG [Producer clientId=jira_prod] Completed connection to node 4. Fetching API versions. (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,407] DEBUG [Producer clientId=jira_prod] Initiating API versions fetch from node 2. (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,407] DEBUG [Producer clientId=jira_prod] Initialize connection to node mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com) for sending metadata request (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,409] DEBUG [Producer clientId=jira_prod] Initiating connection to node mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com) using address mwkafka-prod-01.nyc.xxx.com/10.219.247.25 (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,409] DEBUG Set SASL client state to SEND_APIVERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,409] DEBUG Creating SaslClient: client=choudhab/hostbased@UNIX.XXX.COM;service=kafka;serviceHostname=mwkafka-prod-01.nyc.xxx.com;mechs=[GSSAPI] (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,410] DEBUG Set SASL client state to COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,410] DEBUG Set SASL client state to COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,410] DEBUG Set SASL client state to SEND_HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,410] DEBUG Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,411] DEBUG [Producer clientId=jira_prod] Recorded API versions for node 6: (Produce(0): 0 to 7 [usable: 7], Fetch(1): 0 to 10 [usable: 10], ListOffsets(2): 0 to 5 [usable: 4], Metadata(3): 0 to 7 [usable: 7], LeaderAndIsr(4): 0 to 2 [usable: 1], StopReplica(5): 0 to 1 [usable: 0], UpdateMetadata(6): 0 to 5 [usable: 4], ControlledShutdown(7): 0 to 2 [usable: 1], OffsetCommit(8): 0 to 6 [usable: 6], OffsetFetch(9): 0 to 5 [usable: 5], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0 to 4 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2 [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2 [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3 [usable: 3], DeleteTopics(20): 0 to 3 [usable: 3], DeleteRecords(21): 0 to 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1], OffsetForLeaderEpoch(23): 0 to 2 [usable: 2], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 2 [usable: 2], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1 [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1 [usable: 1], SaslAuthenticate(36): 0 to 1 [usable: 0], CreatePartitions(37): 0 to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1], RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0 to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1], DeleteGroups(42): 0 to 1 [usable: 1], UNKNOWN(43): 0) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,411] DEBUG [Producer clientId=jira_prod] Initiating API versions fetch from node 3. (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,411] DEBUG [Producer clientId=jira_prod] Initiating API versions fetch from node 5. (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,411] DEBUG Added sensor with name node-1.bytes-sent (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,412] DEBUG Added sensor with name node-1.bytes-received (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,412] DEBUG Added sensor with name node-1.latency (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,412] DEBUG [Producer clientId=jira_prod] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 1 (org.apache.kafka.common.network.Selector) [2019-03-05 09:03:04,412] DEBUG Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,413] DEBUG Set SASL client state to INITIAL (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,416] DEBUG Set SASL client state to INTERMEDIATE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,416] DEBUG [Producer clientId=jira_prod] Completed connection to node 1. Fetching API versions. (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,416] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-1,chow-test-0) to node mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,417] DEBUG Set SASL client state to SEND_HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,417] DEBUG Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,417] DEBUG [Producer clientId=jira_prod] Recorded API versions for node 2: (Produce(0): 0 to 7 [usable: 7], Fetch(1): 0 to 10 [usable: 10], ListOffsets(2): 0 to 5 [usable: 4], Metadata(3): 0 to 7 [usable: 7], LeaderAndIsr(4): 0 to 2 [usable: 1], StopReplica(5): 0 to 1 [usable: 0], UpdateMetadata(6): 0 to 5 [usable: 4], ControlledShutdown(7): 0 to 2 [usable: 1], OffsetCommit(8): 0 to 6 [usable: 6], OffsetFetch(9): 0 to 5 [usable: 5], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0 to 4 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2 [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2 [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3 [usable: 3], DeleteTopics(20): 0 to 3 [usable: 3], DeleteRecords(21): 0 to 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1], OffsetForLeaderEpoch(23): 0 to 2 [usable: 2], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 2 [usable: 2], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1 [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1 [usable: 1], SaslAuthenticate(36): 0 to 1 [usable: 0], CreatePartitions(37): 0 to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1], RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0 to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1], DeleteGroups(42): 0 to 1 [usable: 1], UNKNOWN(43): 0) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,418] DEBUG [Producer clientId=jira_prod] Recorded API versions for node 3: (Produce(0): 0 to 7 [usable: 7], Fetch(1): 0 to 10 [usable: 10], ListOffsets(2): 0 to 5 [usable: 4], Metadata(3): 0 to 7 [usable: 7], LeaderAndIsr(4): 0 to 2 [usable: 1], StopReplica(5): 0 to 1 [usable: 0], UpdateMetadata(6): 0 to 5 [usable: 4], ControlledShutdown(7): 0 to 2 [usable: 1], OffsetCommit(8): 0 to 6 [usable: 6], OffsetFetch(9): 0 to 5 [usable: 5], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0 to 4 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2 [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2 [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3 [usable: 3], DeleteTopics(20): 0 to 3 [usable: 3], DeleteRecords(21): 0 to 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1], OffsetForLeaderEpoch(23): 0 to 2 [usable: 2], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 2 [usable: 2], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1 [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1 [usable: 1], SaslAuthenticate(36): 0 to 1 [usable: 0], CreatePartitions(37): 0 to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1], RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0 to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1], DeleteGroups(42): 0 to 1 [usable: 1], UNKNOWN(43): 0) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,418] DEBUG [Producer clientId=jira_prod] Recorded API versions for node 5: (Produce(0): 0 to 7 [usable: 7], Fetch(1): 0 to 10 [usable: 10], ListOffsets(2): 0 to 5 [usable: 4], Metadata(3): 0 to 7 [usable: 7], LeaderAndIsr(4): 0 to 2 [usable: 1], StopReplica(5): 0 to 1 [usable: 0], UpdateMetadata(6): 0 to 5 [usable: 4], ControlledShutdown(7): 0 to 2 [usable: 1], OffsetCommit(8): 0 to 6 [usable: 6], OffsetFetch(9): 0 to 5 [usable: 5], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0 to 4 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2 [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2 [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3 [usable: 3], DeleteTopics(20): 0 to 3 [usable: 3], DeleteRecords(21): 0 to 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1], OffsetForLeaderEpoch(23): 0 to 2 [usable: 2], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 2 [usable: 2], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1 [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1 [usable: 1], SaslAuthenticate(36): 0 to 1 [usable: 0], CreatePartitions(37): 0 to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1], RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0 to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1], DeleteGroups(42): 0 to 1 [usable: 1], UNKNOWN(43): 0) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,422] DEBUG Added sensor with name topic.chow-test-0.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,422] DEBUG Added sensor with name topic.chow-test-0.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,422] DEBUG Added sensor with name topic.chow-test-0.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,422] DEBUG Added sensor with name topic.chow-test-0.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,422] DEBUG Added sensor with name topic.chow-test-0.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,425] DEBUG Set SASL client state to INITIAL (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,425] DEBUG Set SASL client state to INTERMEDIATE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,426] DEBUG Set SASL client state to CLIENT_COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,426] DEBUG Updated cluster metadata version 3 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com), mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com), mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com), mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com), mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com)], partitions = [Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-1, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,426] DEBUG Set SASL client state to COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,426] DEBUG [Producer clientId=jira_prod] Initiating API versions fetch from node 4. (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,427] DEBUG Added sensor with name topic.chow-test-1.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,427] DEBUG Added sensor with name topic.chow-test-1.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,427] DEBUG Added sensor with name topic.chow-test-1.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,427] DEBUG Added sensor with name topic.chow-test-1.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,427] DEBUG Added sensor with name topic.chow-test-1.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,428] DEBUG Set SASL client state to CLIENT_COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,435] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-1,chow-test-2,chow-test-0) to node mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,436] DEBUG Set SASL client state to COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator) [2019-03-05 09:03:04,436] DEBUG [Producer clientId=jira_prod] Recorded API versions for node 4: (Produce(0): 0 to 7 [usable: 7], Fetch(1): 0 to 10 [usable: 10], ListOffsets(2): 0 to 5 [usable: 4], Metadata(3): 0 to 7 [usable: 7], LeaderAndIsr(4): 0 to 2 [usable: 1], StopReplica(5): 0 to 1 [usable: 0], UpdateMetadata(6): 0 to 5 [usable: 4], ControlledShutdown(7): 0 to 2 [usable: 1], OffsetCommit(8): 0 to 6 [usable: 6], OffsetFetch(9): 0 to 5 [usable: 5], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0 to 4 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2 [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2 [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3 [usable: 3], DeleteTopics(20): 0 to 3 [usable: 3], DeleteRecords(21): 0 to 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1], OffsetForLeaderEpoch(23): 0 to 2 [usable: 2], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 2 [usable: 2], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1 [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1 [usable: 1], SaslAuthenticate(36): 0 to 1 [usable: 0], CreatePartitions(37): 0 to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1], RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0 to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1], DeleteGroups(42): 0 to 1 [usable: 1], UNKNOWN(43): 0) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,436] DEBUG [Producer clientId=jira_prod] Initiating API versions fetch from node 1. (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,438] DEBUG Updated cluster metadata version 4 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com), mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com), mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com), mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com), mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com)], partitions = [Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-1, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-2, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,439] DEBUG [Producer clientId=jira_prod] Recorded API versions for node 1: (Produce(0): 0 to 7 [usable: 7], Fetch(1): 0 to 10 [usable: 10], ListOffsets(2): 0 to 5 [usable: 4], Metadata(3): 0 to 7 [usable: 7], LeaderAndIsr(4): 0 to 2 [usable: 1], StopReplica(5): 0 to 1 [usable: 0], UpdateMetadata(6): 0 to 5 [usable: 4], ControlledShutdown(7): 0 to 2 [usable: 1], OffsetCommit(8): 0 to 6 [usable: 6], OffsetFetch(9): 0 to 5 [usable: 5], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0 to 4 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2 [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2 [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3 [usable: 3], DeleteTopics(20): 0 to 3 [usable: 3], DeleteRecords(21): 0 to 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1], OffsetForLeaderEpoch(23): 0 to 2 [usable: 2], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 2 [usable: 2], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1 [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1 [usable: 1], SaslAuthenticate(36): 0 to 1 [usable: 0], CreatePartitions(37): 0 to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1], RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0 to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1], DeleteGroups(42): 0 to 1 [usable: 1], UNKNOWN(43): 0) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,439] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-3,chow-test-1,chow-test-2,chow-test-0) to node mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,439] DEBUG Added sensor with name topic.chow-test-2.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,440] DEBUG Added sensor with name topic.chow-test-2.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,440] DEBUG Added sensor with name topic.chow-test-2.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,440] DEBUG Added sensor with name topic.chow-test-2.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,440] DEBUG Added sensor with name topic.chow-test-2.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,441] DEBUG Updated cluster metadata version 5 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com), mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com), mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com), mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com), mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com)], partitions = [Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-1, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-2, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-3, partition = 0, leader = 6, replicas = [6,4,2], isr = [6,2,4], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,442] DEBUG Added sensor with name topic.chow-test-3.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,442] DEBUG Added sensor with name topic.chow-test-3.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,442] DEBUG Added sensor with name topic.chow-test-3.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,442] DEBUG Added sensor with name topic.chow-test-3.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,442] DEBUG Added sensor with name topic.chow-test-3.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,442] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-3,chow-test-4,chow-test-1,chow-test-2,chow-test-0) to node mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,446] DEBUG Updated cluster metadata version 6 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com), mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com), mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com), mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com), mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com)], partitions = [Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-1, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-4, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-2, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-3, partition = 0, leader = 6, replicas = [6,4,2], isr = [6,2,4], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,446] DEBUG Added sensor with name topic.chow-test-4.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,446] DEBUG Added sensor with name topic.chow-test-4.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,447] DEBUG Added sensor with name topic.chow-test-4.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,447] DEBUG Added sensor with name topic.chow-test-4.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,447] DEBUG Added sensor with name topic.chow-test-4.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,449] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-5,chow-test-3,chow-test-4,chow-test-1,chow-test-2,chow-test-0) to node mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,456] DEBUG Updated cluster metadata version 7 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com), mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com), mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com), mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com), mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com)], partitions = [Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-1, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-4, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-5, partition = 0, leader = 1, replicas = [1,3,5], isr = [1,3,5], offlineReplicas = []), Partition(topic = chow-test-2, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-3, partition = 0, leader = 6, replicas = [6,4,2], isr = [6,2,4], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,456] DEBUG Added sensor with name topic.chow-test-5.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,457] DEBUG Added sensor with name topic.chow-test-5.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,457] DEBUG Added sensor with name topic.chow-test-5.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,457] DEBUG Added sensor with name topic.chow-test-5.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,457] DEBUG Added sensor with name topic.chow-test-5.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,458] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-5,chow-test-6,chow-test-3,chow-test-4,chow-test-1,chow-test-2,chow-test-0) to node mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,460] DEBUG Updated cluster metadata version 8 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com), mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com), mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com), mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com), mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)], partitions = [Partition(topic = chow-test-6, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-4, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-5, partition = 0, leader = 1, replicas = [1,3,5], isr = [1,3,5], offlineReplicas = []), Partition(topic = chow-test-2, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-3, partition = 0, leader = 6, replicas = [6,4,2], isr = [6,2,4], offlineReplicas = []), Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-1, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,460] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-7,chow-test-5,chow-test-6,chow-test-3,chow-test-4,chow-test-1,chow-test-2,chow-test-0) to node mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,461] DEBUG Added sensor with name topic.chow-test-6.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,461] DEBUG Added sensor with name topic.chow-test-6.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,461] DEBUG Added sensor with name topic.chow-test-6.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,461] DEBUG Added sensor with name topic.chow-test-6.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,462] DEBUG Added sensor with name topic.chow-test-6.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,463] DEBUG Updated cluster metadata version 9 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com), mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com), mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com), mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com), mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)], partitions = [Partition(topic = chow-test-6, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-7, partition = 0, leader = 4, replicas = [4,6,1], isr = [6,1,4], offlineReplicas = []), Partition(topic = chow-test-4, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-5, partition = 0, leader = 1, replicas = [1,3,5], isr = [1,3,5], offlineReplicas = []), Partition(topic = chow-test-2, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-3, partition = 0, leader = 6, replicas = [6,4,2], isr = [6,2,4], offlineReplicas = []), Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-1, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,464] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-7,chow-test-8,chow-test-5,chow-test-6,chow-test-3,chow-test-4,chow-test-1,chow-test-2,chow-test-0) to node mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,464] DEBUG Added sensor with name topic.chow-test-7.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,465] DEBUG Added sensor with name topic.chow-test-7.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,465] DEBUG Added sensor with name topic.chow-test-7.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,465] DEBUG Added sensor with name topic.chow-test-7.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,465] DEBUG Added sensor with name topic.chow-test-7.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,466] DEBUG Updated cluster metadata version 10 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com), mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com), mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com), mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com), mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com)], partitions = [Partition(topic = chow-test-8, partition = 0, leader = 2, replicas = [2,3,5], isr = [2,3,5], offlineReplicas = []), Partition(topic = chow-test-6, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-7, partition = 0, leader = 4, replicas = [4,6,1], isr = [6,1,4], offlineReplicas = []), Partition(topic = chow-test-4, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-5, partition = 0, leader = 1, replicas = [1,3,5], isr = [1,3,5], offlineReplicas = []), Partition(topic = chow-test-2, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-3, partition = 0, leader = 6, replicas = [6,4,2], isr = [6,2,4], offlineReplicas = []), Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-1, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,466] DEBUG Added sensor with name topic.chow-test-8.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,467] DEBUG Added sensor with name topic.chow-test-8.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,467] DEBUG Added sensor with name topic.chow-test-8.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,467] DEBUG Added sensor with name topic.chow-test-8.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,467] DEBUG Added sensor with name topic.chow-test-8.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,467] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-9,chow-test-7,chow-test-8,chow-test-5,chow-test-6,chow-test-3,chow-test-4,chow-test-1,chow-test-2,chow-test-0) to node mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,470] DEBUG Updated cluster metadata version 11 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com), mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com), mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com), mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com), mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com)], partitions = [Partition(topic = chow-test-8, partition = 0, leader = 2, replicas = [2,3,5], isr = [2,3,5], offlineReplicas = []), Partition(topic = chow-test-9, partition = 0, leader = 5, replicas = [5,4,2], isr = [5,2,4], offlineReplicas = []), Partition(topic = chow-test-6, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-7, partition = 0, leader = 4, replicas = [4,6,1], isr = [6,1,4], offlineReplicas = []), Partition(topic = chow-test-4, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-5, partition = 0, leader = 1, replicas = [1,3,5], isr = [1,3,5], offlineReplicas = []), Partition(topic = chow-test-2, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-3, partition = 0, leader = 6, replicas = [6,4,2], isr = [6,2,4], offlineReplicas = []), Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-1, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,471] DEBUG Added sensor with name topic.chow-test-9.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,471] DEBUG Added sensor with name topic.chow-test-9.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,471] DEBUG Added sensor with name topic.chow-test-9.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,471] DEBUG Added sensor with name topic.chow-test-9.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,471] DEBUG Added sensor with name topic.chow-test-9.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,472] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-9,chow-test-7,chow-test-8,chow-test-5,chow-test-10,chow-test-6,chow-test-3,chow-test-4,chow-test-1,chow-test-2,chow-test-0) to node mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,477] DEBUG Updated cluster metadata version 12 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com), mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com), mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com), mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com), mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com)], partitions = [Partition(topic = chow-test-8, partition = 0, leader = 2, replicas = [2,3,5], isr = [2,3,5], offlineReplicas = []), Partition(topic = chow-test-9, partition = 0, leader = 5, replicas = [5,4,2], isr = [5,2,4], offlineReplicas = []), Partition(topic = chow-test-6, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-7, partition = 0, leader = 4, replicas = [4,6,1], isr = [6,1,4], offlineReplicas = []), Partition(topic = chow-test-4, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-10, partition = 0, leader = 2, replicas = [2,6,3], isr = [2,6,3], offlineReplicas = []), Partition(topic = chow-test-5, partition = 0, leader = 1, replicas = [1,3,5], isr = [1,3,5], offlineReplicas = []), Partition(topic = chow-test-2, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-3, partition = 0, leader = 6, replicas = [6,4,2], isr = [6,2,4], offlineReplicas = []), Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-1, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,477] DEBUG Added sensor with name topic.chow-test-10.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,478] DEBUG Added sensor with name topic.chow-test-10.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,478] DEBUG Added sensor with name topic.chow-test-10.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,478] DEBUG Added sensor with name topic.chow-test-10.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,478] DEBUG Added sensor with name topic.chow-test-10.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,478] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-9,chow-test-7,chow-test-8,chow-test-5,chow-test-10,chow-test-6,chow-test-11,chow-test-3,chow-test-4,chow-test-1,chow-test-2,chow-test-0) to node mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,480] DEBUG Updated cluster metadata version 13 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com), mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com), mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com), mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com), mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com)], partitions = [Partition(topic = chow-test-8, partition = 0, leader = 2, replicas = [2,3,5], isr = [2,3,5], offlineReplicas = []), Partition(topic = chow-test-9, partition = 0, leader = 5, replicas = [5,4,2], isr = [5,2,4], offlineReplicas = []), Partition(topic = chow-test-11, partition = 0, leader = 4, replicas = [4,6,1], isr = [6,1,4], offlineReplicas = []), Partition(topic = chow-test-6, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-7, partition = 0, leader = 4, replicas = [4,6,1], isr = [6,1,4], offlineReplicas = []), Partition(topic = chow-test-4, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-10, partition = 0, leader = 2, replicas = [2,6,3], isr = [2,6,3], offlineReplicas = []), Partition(topic = chow-test-5, partition = 0, leader = 1, replicas = [1,3,5], isr = [1,3,5], offlineReplicas = []), Partition(topic = chow-test-2, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-3, partition = 0, leader = 6, replicas = [6,4,2], isr = [6,2,4], offlineReplicas = []), Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-1, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,481] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-9,chow-test-7,chow-test-12,chow-test-8,chow-test-5,chow-test-10,chow-test-6,chow-test-11,chow-test-3,chow-test-4,chow-test-1,chow-test-2,chow-test-0) to node mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,482] DEBUG Added sensor with name topic.chow-test-11.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,482] DEBUG Added sensor with name topic.chow-test-11.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,482] DEBUG Added sensor with name topic.chow-test-11.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,482] DEBUG Added sensor with name topic.chow-test-11.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,482] DEBUG Added sensor with name topic.chow-test-11.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,484] DEBUG Updated cluster metadata version 14 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com), mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com), mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com), mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com), mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com)], partitions = [Partition(topic = chow-test-8, partition = 0, leader = 2, replicas = [2,3,5], isr = [2,3,5], offlineReplicas = []), Partition(topic = chow-test-9, partition = 0, leader = 5, replicas = [5,4,2], isr = [5,2,4], offlineReplicas = []), Partition(topic = chow-test-11, partition = 0, leader = 4, replicas = [4,6,1], isr = [6,1,4], offlineReplicas = []), Partition(topic = chow-test-6, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-12, partition = 0, leader = 4, replicas = [4,1,5], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-7, partition = 0, leader = 4, replicas = [4,6,1], isr = [6,1,4], offlineReplicas = []), Partition(topic = chow-test-4, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-10, partition = 0, leader = 2, replicas = [2,6,3], isr = [2,6,3], offlineReplicas = []), Partition(topic = chow-test-5, partition = 0, leader = 1, replicas = [1,3,5], isr = [1,3,5], offlineReplicas = []), Partition(topic = chow-test-2, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-3, partition = 0, leader = 6, replicas = [6,4,2], isr = [6,2,4], offlineReplicas = []), Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-1, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,484] DEBUG Added sensor with name topic.chow-test-12.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,484] DEBUG Added sensor with name topic.chow-test-12.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,485] DEBUG Added sensor with name topic.chow-test-12.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,485] DEBUG Added sensor with name topic.chow-test-12.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,485] DEBUG Added sensor with name topic.chow-test-12.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,485] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-9,chow-test-7,chow-test-12,chow-test-8,chow-test-13,chow-test-5,chow-test-10,chow-test-6,chow-test-11,chow-test-3,chow-test-4,chow-test-1,chow-test-2,chow-test-0) to node mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,488] DEBUG Updated cluster metadata version 15 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com), mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com), mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com), mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com), mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com)], partitions = [Partition(topic = chow-test-8, partition = 0, leader = 2, replicas = [2,3,5], isr = [2,3,5], offlineReplicas = []), Partition(topic = chow-test-13, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-9, partition = 0, leader = 5, replicas = [5,4,2], isr = [5,2,4], offlineReplicas = []), Partition(topic = chow-test-11, partition = 0, leader = 4, replicas = [4,6,1], isr = [6,1,4], offlineReplicas = []), Partition(topic = chow-test-6, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-12, partition = 0, leader = 4, replicas = [4,1,5], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-7, partition = 0, leader = 4, replicas = [4,6,1], isr = [6,1,4], offlineReplicas = []), Partition(topic = chow-test-4, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-10, partition = 0, leader = 2, replicas = [2,6,3], isr = [2,6,3], offlineReplicas = []), Partition(topic = chow-test-5, partition = 0, leader = 1, replicas = [1,3,5], isr = [1,3,5], offlineReplicas = []), Partition(topic = chow-test-2, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-3, partition = 0, leader = 6, replicas = [6,4,2], isr = [6,2,4], offlineReplicas = []), Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-1, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,488] DEBUG Added sensor with name topic.chow-test-13.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,488] DEBUG Added sensor with name topic.chow-test-13.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,488] DEBUG Added sensor with name topic.chow-test-13.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,488] DEBUG Added sensor with name topic.chow-test-13.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,489] DEBUG Added sensor with name topic.chow-test-13.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,489] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-9,chow-test-14,chow-test-7,chow-test-12,chow-test-8,chow-test-13,chow-test-5,chow-test-10,chow-test-6,chow-test-11,chow-test-3,chow-test-4,chow-test-1,chow-test-2,chow-test-0) to node mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,491] DEBUG Updated cluster metadata version 16 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com), mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com), mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com), mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com), mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com)], partitions = [Partition(topic = chow-test-8, partition = 0, leader = 2, replicas = [2,3,5], isr = [2,3,5], offlineReplicas = []), Partition(topic = chow-test-13, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-9, partition = 0, leader = 5, replicas = [5,4,2], isr = [5,2,4], offlineReplicas = []), Partition(topic = chow-test-14, partition = 0, leader = 1, replicas = [1,3,5], isr = [1,3,5], offlineReplicas = []), Partition(topic = chow-test-11, partition = 0, leader = 4, replicas = [4,6,1], isr = [6,1,4], offlineReplicas = []), Partition(topic = chow-test-6, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-12, partition = 0, leader = 4, replicas = [4,1,5], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-7, partition = 0, leader = 4, replicas = [4,6,1], isr = [6,1,4], offlineReplicas = []), Partition(topic = chow-test-4, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-10, partition = 0, leader = 2, replicas = [2,6,3], isr = [2,6,3], offlineReplicas = []), Partition(topic = chow-test-5, partition = 0, leader = 1, replicas = [1,3,5], isr = [1,3,5], offlineReplicas = []), Partition(topic = chow-test-2, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-3, partition = 0, leader = 6, replicas = [6,4,2], isr = [6,2,4], offlineReplicas = []), Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-1, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,492] DEBUG Added sensor with name topic.chow-test-14.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,492] DEBUG Added sensor with name topic.chow-test-14.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,492] DEBUG Added sensor with name topic.chow-test-14.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,492] DEBUG Added sensor with name topic.chow-test-14.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,492] DEBUG Added sensor with name topic.chow-test-14.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,498] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-9,chow-test-14,chow-test-15,chow-test-7,chow-test-12,chow-test-8,chow-test-13,chow-test-5,chow-test-10,chow-test-6,chow-test-11,chow-test-3,chow-test-4,chow-test-1,chow-test-2,chow-test-0) to node mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,500] DEBUG Updated cluster metadata version 17 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com), mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com), mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com), mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com), mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)], partitions = [Partition(topic = chow-test-15, partition = 0, leader = 2, replicas = [2,3,5], isr = [2,3,5], offlineReplicas = []), Partition(topic = chow-test-8, partition = 0, leader = 2, replicas = [2,3,5], isr = [2,3,5], offlineReplicas = []), Partition(topic = chow-test-13, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-9, partition = 0, leader = 5, replicas = [5,4,2], isr = [5,2,4], offlineReplicas = []), Partition(topic = chow-test-14, partition = 0, leader = 1, replicas = [1,3,5], isr = [1,3,5], offlineReplicas = []), Partition(topic = chow-test-11, partition = 0, leader = 4, replicas = [4,6,1], isr = [6,1,4], offlineReplicas = []), Partition(topic = chow-test-6, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-12, partition = 0, leader = 4, replicas = [4,1,5], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-7, partition = 0, leader = 4, replicas = [4,6,1], isr = [6,1,4], offlineReplicas = []), Partition(topic = chow-test-4, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-10, partition = 0, leader = 2, replicas = [2,6,3], isr = [2,6,3], offlineReplicas = []), Partition(topic = chow-test-5, partition = 0, leader = 1, replicas = [1,3,5], isr = [1,3,5], offlineReplicas = []), Partition(topic = chow-test-2, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-3, partition = 0, leader = 6, replicas = [6,4,2], isr = [6,2,4], offlineReplicas = []), Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-1, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,501] DEBUG Added sensor with name topic.chow-test-15.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,501] DEBUG Added sensor with name topic.chow-test-15.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,501] DEBUG Added sensor with name topic.chow-test-15.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,501] DEBUG Added sensor with name topic.chow-test-15.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,501] DEBUG Added sensor with name topic.chow-test-15.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,501] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-16,chow-test-9,chow-test-14,chow-test-15,chow-test-7,chow-test-12,chow-test-8,chow-test-13,chow-test-5,chow-test-10,chow-test-6,chow-test-11,chow-test-3,chow-test-4,chow-test-1,chow-test-2,chow-test-0) to node mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,505] DEBUG Updated cluster metadata version 18 to Cluster(id = bIfwPQCgTry9_1SkT9W-1A, nodes = [mwkafka-prod-01.dr.xxx.com:9092 (id: 3 rack: dr.xxx.com), mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com), mwkafka-prod-02.nyc.xxx.com:9092 (id: 2 rack: nyc.xxx.com), mwkafka-prod-01.tbd.xxx.com:9092 (id: 5 rack: tbd.xxx.com), mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com), mwkafka-prod-02.dr.xxx.com:9092 (id: 4 rack: dr.xxx.com)], partitions = [Partition(topic = chow-test-15, partition = 0, leader = 2, replicas = [2,3,5], isr = [2,3,5], offlineReplicas = []), Partition(topic = chow-test-16, partition = 0, leader = 3, replicas = [3,5,2], isr = [3,5,2], offlineReplicas = []), Partition(topic = chow-test-8, partition = 0, leader = 2, replicas = [2,3,5], isr = [2,3,5], offlineReplicas = []), Partition(topic = chow-test-13, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-9, partition = 0, leader = 5, replicas = [5,4,2], isr = [5,2,4], offlineReplicas = []), Partition(topic = chow-test-14, partition = 0, leader = 1, replicas = [1,3,5], isr = [1,3,5], offlineReplicas = []), Partition(topic = chow-test-11, partition = 0, leader = 4, replicas = [4,6,1], isr = [6,1,4], offlineReplicas = []), Partition(topic = chow-test-6, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-12, partition = 0, leader = 4, replicas = [4,1,5], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-7, partition = 0, leader = 4, replicas = [4,6,1], isr = [6,1,4], offlineReplicas = []), Partition(topic = chow-test-4, partition = 0, leader = 5, replicas = [5,3,1], isr = [5,3,1], offlineReplicas = []), Partition(topic = chow-test-10, partition = 0, leader = 2, replicas = [2,6,3], isr = [2,6,3], offlineReplicas = []), Partition(topic = chow-test-5, partition = 0, leader = 1, replicas = [1,3,5], isr = [1,3,5], offlineReplicas = []), Partition(topic = chow-test-2, partition = 0, leader = 1, replicas = [1,5,4], isr = [1,5,4], offlineReplicas = []), Partition(topic = chow-test-3, partition = 0, leader = 6, replicas = [6,4,2], isr = [6,2,4], offlineReplicas = []), Partition(topic = chow-test-0, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = []), Partition(topic = chow-test-1, partition = 0, leader = 6, replicas = [6,3,1], isr = [6,3,1], offlineReplicas = [])], controller = mwkafka-prod-01.nyc.xxx.com:9092 (id: 1 rack: nyc.xxx.com)) (org.apache.kafka.clients.Metadata) [2019-03-05 09:03:04,505] DEBUG Added sensor with name topic.chow-test-16.records-per-batch (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,505] DEBUG Added sensor with name topic.chow-test-16.bytes (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,505] DEBUG Added sensor with name topic.chow-test-16.compression-rate (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,505] DEBUG Added sensor with name topic.chow-test-16.record-retries (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,505] DEBUG Added sensor with name topic.chow-test-16.record-errors (org.apache.kafka.common.metrics.Metrics) [2019-03-05 09:03:04,506] DEBUG [Producer clientId=jira_prod] Sending metadata request (type=MetadataRequest, topics=chow-test-16,chow-test-17,chow-test-9,chow-test-14,chow-test-15,chow-test-7,chow-test-12,chow-test-8,chow-test-13,chow-test-5,chow-test-10,chow-test-6,chow-test-11,chow-test-3,chow-test-4,chow-test-1,chow-test-2,chow-test-0) to node mwkafka-prod-02.tbd.xxx.com:9092 (id: 6 rack: tbd.xxx.com) (org.apache.kafka.clients.NetworkClient) [2019-03-05 09:03:04,507] ERROR Uncaught exception in thread 'kafka-producer-network-thread | jira_prod': (org.apache.kafka.common.utils.KafkaThread) xxx.common.util.ApplicationDeath: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request. at xxx.kafka.test.KafkaTestProducer$ProducerCallback.onCompletion(KafkaTestProducer.java:475) at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1304) at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:230) at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:196) at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:717) at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:685) at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:635) at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:557) at org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:74) at org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:786) at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109) at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:557) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.