Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4003

Followers are dropped for being too far behind, yet are less than 30 messages / 10k bytes behind.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.9.0.1
    • None
    • core
    • None

    Description

      1. Settings are at defaults other than the following
        replica.fetch.max.bytes=52428800
        num.replica.fetchers=8
        num.network.threads=128

      Followers are dropped for being too far behind, yet are less than 30 messages / 10k bytes behind. The issue becomes more apparent during periods of high load, though is not exclusive to high load periods.

      We enabled the log4j plugin and turned on DEBUG (for 60s) for
      kafka.cluster.Partition
      kafka.server.ReplicaManager

      The attached DEBUG log data is excerpted for topic,partition 'contacts-pub-sub,7' and shows that reads are happening. However the reads never read up to the log end offset of the Leader Replica, and thus the Replica.lastCaughtUpTimeMsUnderlying field is not updated for extended periods of time, which gets the follower dropped. Not too long after being dropped, the follower will get picked up again.

      [2016-07-28 13:31:36,608] DEBUG [Replica Manager on Broker 157]: Recording follower broker 158 log read results: Map([email-raw-sparkpost-events-sox-signed-unknown,9] -> Fetch Data: [FetchDataInfo(1434 [1286 : 104914],ByteBufferMessageSet())], HW: [1434], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [email-events-eventtracking,15] -> Fetch Data: [FetchDataInfo(109074743 [108253610 : 830497723],ByteBufferMessageSet())], HW: [109074743], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [oauth-authorization,26] -> Fetch Data: [FetchDataInfo(5468 [5400 : 5469],ByteBufferMessageSet())], HW: [5468], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [eventtracking-event-callbacks-c,14] -> Fetch Data: [FetchDataInfo(392319869 [391972118 : 470595480],ByteBufferMessageSet())], HW: [392319869], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [emailvalidation-strikeiron-responses,8] -> Fetch Data: [FetchDataInfo(3547001 [3527594 : 19771984],ByteBufferMessageSet())], HW: [3547001], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [usage-events-raw,1] -> Fetch Data: [FetchDataInfo(316637965 [316090773 : 831265168],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 0, crc = 286015273, key = java.nio.HeapByteBuffer[pos=0 lim=67 cap=2331], payload = java.nio.HeapByteBuffer[pos=0 lim=2260 cap=2260]),316637965), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1018327823, key = java.nio.HeapByteBuffer[pos=0 lim=91 cap=1546], payload = java.nio.HeapByteBuffer[pos=0 lim=1451 cap=1451]),316637966), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2467233135, key = java.nio.HeapByteBuffer[pos=0 lim=65 cap=1429], payload = java.nio.HeapByteBuffer[pos=0 lim=1360 cap=1360]),316637967)))], HW: [316637962], readSize: [50000000], isReadFromLogEnd: [false], error: [None], [contacts-pub-sub,7] -> Fetch Data: [FetchDataInfo(2835905113 [2834198288 : 966349167],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1940228036, key = java.nio.HeapByteBuffer[pos=0 lim=12 cap=410], payload = java.nio.HeapByteBuffer[pos=0 lim=394 cap=394]),2835905113), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1613735028, key = java.nio.HeapByteBuffer[pos=0 lim=14 cap=576], payload = java.nio.HeapByteBuffer[pos=0 lim=558 cap=558]),2835905114), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1573356394, key = java.nio.HeapByteBuffer[pos=0 lim=16 cap=844], payload = java.nio.HeapByteBuffer[pos=0 lim=824 cap=824]),2835905115), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3816197841, key = java.nio.HeapByteBuffer[pos=0 lim=14 cap=1326], payload = java.nio.HeapByteBuffer[pos=0 lim=1308 cap=1308]),2835905116), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2906657432, key = java.nio.HeapByteBuffer[pos=0 lim=15 cap=243], payload = java.nio.HeapByteBuffer[pos=0 lim=224 cap=224]),2835905117), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1616776650, key = java.nio.HeapByteBuffer[pos=0 lim=16 cap=845], payload = java.nio.HeapByteBuffer[pos=0 lim=825 cap=825]),2835905118), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2182693743, key = java.nio.HeapByteBuffer[pos=0 lim=15 cap=778], payload = java.nio.HeapByteBuffer[pos=0 lim=759 cap=759]),2835905119), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3254937137, key = java.nio.HeapByteBuffer[pos=0 lim=15 cap=1375], payload = java.nio.HeapByteBuffer[pos=0 lim=1356 cap=1356]),2835905120), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1434287683, key = java.nio.HeapByteBuffer[pos=0 lim=14 cap=412], payload = java.nio.HeapByteBuffer[pos=0 lim=394 cap=394]),2835905121)))], HW: [2835905103], readSize: [50000000], isReadFromLogEnd: [false], error: [None], [email-events-sox-verified,11] -> Fetch Data: [FetchDataInfo(1149863327 [1149653464 : 125554894],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 2, crc = 758891241, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1249 cap=1249]),1149863327), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 2037653886, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1218 cap=1218]),1149863328), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 1564011888, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1219 cap=1219]),1149863329), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 2958390241, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1271 cap=1271]),1149863330), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 704061401, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1312 cap=1312]),1149863331), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 362204766, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1251 cap=1251]),1149863332), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 2874177783, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1212 cap=1212]),1149863333), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 1728470591, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1204 cap=1204]),1149863334), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 4127052822, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=16617 cap=16617]),1149863377)))], HW: [1149863270], readSize: [50000000], isReadFromLogEnd: [false], error: [None], [email-events-sox-verified,3] -> Fetch Data: [FetchDataInfo(1149891315 [1149636809 : 160829183],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 2, crc = 50411905, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1183 cap=1183]),1149891315), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 3321837456, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1225 cap=1225]),1149891316), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 4128609644, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1219 cap=1219]),1149891317), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 3262264350, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1272 cap=1272]),1149891318), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 4172597564, key = java.nio.HeapByteBuffer[pos=0 lim=36 cap=685], payload = java.nio.HeapByteBuffer[pos=0 lim=645 cap=645]),1149891319), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 672906920, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1214 cap=1214]),1149891320), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 4044991181, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=15423 cap=15423]),1149891360), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 3182323659, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1232 cap=1232]),1149891361)))], HW: [1149891260], readSize: [50000000], isReadFromLogEnd: [false], error: [None]) (kafka.server.ReplicaManager)
      [2016-07-28 13:31:36,616] DEBUG Partition [contacts-pub-sub,7] on broker 157: Skipping update high watermark since Old hw 2835905103 [2834198288 : 966344226] is larger than new hw 2835905103 [2834198288 : 966344226] for partition [contacts-pub-sub,7]. All leo's are 2835905123 [2834198288 : 966356596],2835905116 [2834198288 : 966351063],2835905103 [2834198288 : 966344226] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,616] DEBUG Partition [contacts-pub-sub,7] on broker 157: Skipping update high watermark since Old hw 2835905103 [2834198288 : 966344226] is larger than new hw 2835905103 [2834198288 : 966344226] for partition [contacts-pub-sub,7]. All leo's are 2835905123 [2834198288 : 966356596],2835905116 [2834198288 : 966351063],2835905103 [2834198288 : 966344226] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,616] DEBUG Partition [contacts-pub-sub,7] on broker 157: Skipping update high watermark since Old hw 2835905103 [2834198288 : 966344226] is larger than new hw 2835905103 [2834198288 : 966344226] for partition [contacts-pub-sub,7]. All leo's are 2835905124 [2834198288 : 966357019],2835905116 [2834198288 : 966351063],2835905103 [2834198288 : 966344226] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,616] DEBUG Partition [contacts-pub-sub,7] on broker 157: Recorded replica 160 log end offset (LEO) position 2835905116 for partition [contacts-pub-sub,7]. (kafka.cluster.Partition)
      [2016-07-28 13:31:36,618] DEBUG Partition [contacts-pub-sub,7] on broker 157: Skipping update high watermark since Old hw 2835905103 [2834198288 : 966344226] is larger than new hw 2835905103 [2834198288 : 966344226] for partition [contacts-pub-sub,7]. All leo's are 2835905125 [2834198288 : 966358120],2835905116 [2834198288 : 966351063],2835905103 [2834198288 : 966344226] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,618] DEBUG Partition [contacts-pub-sub,7] on broker 157: Skipping update high watermark since Old hw 2835905103 [2834198288 : 966344226] is larger than new hw 2835905103 [2834198288 : 966344226] for partition [contacts-pub-sub,7]. All leo's are 2835905126 [2834198288 : 966358532],2835905116 [2834198288 : 966351063],2835905103 [2834198288 : 966344226] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,623] DEBUG Partition [contacts-pub-sub,7] on broker 157: Skipping update high watermark since Old hw 2835905103 [2834198288 : 966344226] is larger than new hw 2835905103 [2834198288 : 966344226] for partition [contacts-pub-sub,7]. All leo's are 2835905127 [2834198288 : 966358952],2835905116 [2834198288 : 966351063],2835905103 [2834198288 : 966344226] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,626] DEBUG Partition [contacts-pub-sub,7] on broker 157: High watermark for partition [contacts-pub-sub,7] updated to 2835905113 [2834198288 : 966349167] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,642] DEBUG Partition [contacts-pub-sub,7] on broker 157: Skipping update high watermark since Old hw 2835905113 [2834198288 : 966349167] is larger than new hw 2835905113 [2834198288 : 966349167] for partition [contacts-pub-sub,7]. All leo's are 2835905129 [2834198288 : 966359797],2835905116 [2834198288 : 966351063],2835905113 [2834198288 : 966349167] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,642] DEBUG Partition [contacts-pub-sub,7] on broker 157: Skipping update high watermark since Old hw 2835905113 [2834198288 : 966349167] is larger than new hw 2835905113 [2834198288 : 966349167] for partition [contacts-pub-sub,7]. All leo's are 2835905129 [2834198288 : 966359797],2835905116 [2834198288 : 966351063],2835905113 [2834198288 : 966349167] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,645] DEBUG Partition [contacts-pub-sub,7] on broker 157: Recorded replica 158 log end offset (LEO) position 2835905113 for partition [contacts-pub-sub,7]. (kafka.cluster.Partition)
      [2016-07-28 13:31:36,647] DEBUG Partition [contacts-pub-sub,7] on broker 157: Skipping update high watermark since Old hw 2835905113 [2834198288 : 966349167] is larger than new hw 2835905113 [2834198288 : 966349167] for partition [contacts-pub-sub,7]. All leo's are 2835905130 [2834198288 : 966360310],2835905116 [2834198288 : 966351063],2835905113 [2834198288 : 966349167] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,656] DEBUG Partition [contacts-pub-sub,7] on broker 157: Lagging replicas for partition [contacts-pub-sub,7] are 160,158 (kafka.cluster.Partition)
      [2016-07-28 13:31:36,657] INFO Partition [contacts-pub-sub,7] on broker 157: Shrinking ISR for partition [contacts-pub-sub,7] from 157,160,158 to 157 (kafka.cluster.Partition)
      [2016-07-28 13:31:36,659] DEBUG Partition [contacts-pub-sub,7] on broker 157: High watermark for partition [contacts-pub-sub,7] updated to 2835905130 [2834198288 : 966360310] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,662] DEBUG [Replica Manager on Broker 157]: Recording follower broker 160 log read results: Map([biden-rss-events,10] -> Fetch Data: [FetchDataInfo(2772927 [2692556 : 27087325],ByteBufferMessageSet())], HW: [2772927], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [cps-es-indexer-a,28] -> Fetch Data: [FetchDataInfo(519301 [496185 : 68076836],ByteBufferMessageSet())], HW: [519301], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [billing-amendment-applied-notifications,7] -> Fetch Data: [FetchDataInfo(2557 [2533 : 2856],ByteBufferMessageSet())], HW: [2557], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [usage-events-raw,1] -> Fetch Data: [FetchDataInfo(316637968 [316090773 : 831270540],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1788356129, key = java.nio.HeapByteBuffer[pos=0 lim=44 cap=956], payload = java.nio.HeapByteBuffer[pos=0 lim=908 cap=908]),316637968), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1996475757, key = java.nio.HeapByteBuffer[pos=0 lim=44 cap=956], payload = java.nio.HeapByteBuffer[pos=0 lim=908 cap=908]),316637969)))], HW: [316637965], readSize: [50000000], isReadFromLogEnd: [false], error: [None], [contacts-pub-sub,7] -> Fetch Data: [FetchDataInfo(2835905122 [2834198288 : 966356174],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3917496555, key = java.nio.HeapByteBuffer[pos=0 lim=14 cap=400], payload = java.nio.HeapByteBuffer[pos=0 lim=382 cap=382]),2835905122), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3390031010, key = java.nio.HeapByteBuffer[pos=0 lim=16 cap=401], payload = java.nio.HeapByteBuffer[pos=0 lim=381 cap=381]),2835905123), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2641803275, key = java.nio.HeapByteBuffer[pos=0 lim=14 cap=1079], payload = java.nio.HeapByteBuffer[pos=0 lim=1061 cap=1061]),2835905124), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 217385418, key = java.nio.HeapByteBuffer[pos=0 lim=16 cap=390], payload = java.nio.HeapByteBuffer[pos=0 lim=370 cap=370]),2835905125), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3321979137, key = java.nio.HeapByteBuffer[pos=0 lim=14 cap=398], payload = java.nio.HeapByteBuffer[pos=0 lim=380 cap=380]),2835905126), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3220348147, key = java.nio.HeapByteBuffer[pos=0 lim=15 cap=401], payload = java.nio.HeapByteBuffer[pos=0 lim=382 cap=382]),2835905127), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2893466760, key = java.nio.HeapByteBuffer[pos=0 lim=14 cap=400], payload = java.nio.HeapByteBuffer[pos=0 lim=382 cap=382]),2835905128), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2397577902, key = java.nio.HeapByteBuffer[pos=0 lim=12 cap=491], payload = java.nio.HeapByteBuffer[pos=0 lim=475 cap=475]),2835905129)))], HW: [2835905130], readSize: [50000000], isReadFromLogEnd: [false], error: [None], [email-events-sox-verified,3] -> Fetch Data: [FetchDataInfo(1149891362 [1149636809 : 160852840],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 2, crc = 712083310, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1244 cap=1244]),1149891362)))], HW: [1149891315], readSize: [50000000], isReadFromLogEnd: [false], error: [None]) (kafka.server.ReplicaManager)
      [2016-07-28 13:31:36,671] DEBUG Partition [contacts-pub-sub,7] on broker 157: Skipping update high watermark since Old hw 2835905130 [2834198288 : 966360310] is larger than new hw 2835905130 [2834198288 : 966360310] for partition [contacts-pub-sub,7]. All leo's are 2835905130 [2834198288 : 966360310] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,671] DEBUG Partition [contacts-pub-sub,7] on broker 157: Recorded replica 160 log end offset (LEO) position 2835905122 for partition [contacts-pub-sub,7]. (kafka.cluster.Partition)
      [2016-07-28 13:31:36,691] ERROR [Replica Manager on Broker 157]: Error processing append operation on partition [contacts-pub-sub,7] (kafka.server.ReplicaManager)
      kafka.common.NotEnoughReplicasException: Number of insync replicas for partition [contacts-pub-sub,7] is [1], below required minimum [2]
      [2016-07-28 13:31:36,698] ERROR [Replica Manager on Broker 157]: Error processing append operation on partition [contacts-pub-sub,7] (kafka.server.ReplicaManager)
      kafka.common.NotEnoughReplicasException: Number of insync replicas for partition [contacts-pub-sub,7] is [1], below required minimum [2]
      [2016-07-28 13:31:36,701] ERROR [Replica Manager on Broker 157]: Error processing append operation on partition [contacts-pub-sub,7] (kafka.server.ReplicaManager)
      kafka.common.NotEnoughReplicasException: Number of insync replicas for partition [contacts-pub-sub,7] is [1], below required minimum [2]
      [2016-07-28 13:31:36,710] ERROR [Replica Manager on Broker 157]: Error processing append operation on partition [contacts-pub-sub,7] (kafka.server.ReplicaManager)
      kafka.common.NotEnoughReplicasException: Number of insync replicas for partition [contacts-pub-sub,7] is [1], below required minimum [2]
      [2016-07-28 13:31:36,709] DEBUG [Replica Manager on Broker 157]: Recording follower broker 158 log read results: Map([email-raw-sparkpost-events-sox-signed-unknown,9] -> Fetch Data: [FetchDataInfo(1434 [1286 : 104914],ByteBufferMessageSet())], HW: [1434], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [email-events-eventtracking,15] -> Fetch Data: [FetchDataInfo(109074743 [108253610 : 830497723],ByteBufferMessageSet())], HW: [109074743], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [oauth-authorization,26] -> Fetch Data: [FetchDataInfo(5468 [5400 : 5469],ByteBufferMessageSet())], HW: [5468], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [eventtracking-event-callbacks-c,14] -> Fetch Data: [FetchDataInfo(392319869 [391972118 : 470595480],ByteBufferMessageSet())], HW: [392319869], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [emailvalidation-strikeiron-responses,8] -> Fetch Data: [FetchDataInfo(3547001 [3527594 : 19771984],ByteBufferMessageSet())], HW: [3547001], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [usage-events-raw,1] -> Fetch Data: [FetchDataInfo(316637968 [316090773 : 831270540],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1788356129, key = java.nio.HeapByteBuffer[pos=0 lim=44 cap=956], payload = java.nio.HeapByteBuffer[pos=0 lim=908 cap=908]),316637968), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 1996475757, key = java.nio.HeapByteBuffer[pos=0 lim=44 cap=956], payload = java.nio.HeapByteBuffer[pos=0 lim=908 cap=908]),316637969)))], HW: [316637965], readSize: [50000000], isReadFromLogEnd: [false], error: [None], [contacts-pub-sub,7] -> Fetch Data: [FetchDataInfo(2835905122 [2834198288 : 966356174],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3917496555, key = java.nio.HeapByteBuffer[pos=0 lim=14 cap=400], payload = java.nio.HeapByteBuffer[pos=0 lim=382 cap=382]),2835905122), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3390031010, key = java.nio.HeapByteBuffer[pos=0 lim=16 cap=401], payload = java.nio.HeapByteBuffer[pos=0 lim=381 cap=381]),2835905123), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2641803275, key = java.nio.HeapByteBuffer[pos=0 lim=14 cap=1079], payload = java.nio.HeapByteBuffer[pos=0 lim=1061 cap=1061]),2835905124), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 217385418, key = java.nio.HeapByteBuffer[pos=0 lim=16 cap=390], payload = java.nio.HeapByteBuffer[pos=0 lim=370 cap=370]),2835905125), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3321979137, key = java.nio.HeapByteBuffer[pos=0 lim=14 cap=398], payload = java.nio.HeapByteBuffer[pos=0 lim=380 cap=380]),2835905126), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 3220348147, key = java.nio.HeapByteBuffer[pos=0 lim=15 cap=401], payload = java.nio.HeapByteBuffer[pos=0 lim=382 cap=382]),2835905127), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2893466760, key = java.nio.HeapByteBuffer[pos=0 lim=14 cap=400], payload = java.nio.HeapByteBuffer[pos=0 lim=382 cap=382]),2835905128), MessageAndOffset(Message(magic = 0, attributes = 0, crc = 2397577902, key = java.nio.HeapByteBuffer[pos=0 lim=12 cap=491], payload = java.nio.HeapByteBuffer[pos=0 lim=475 cap=475]),2835905129)))], HW: [2835905130], readSize: [50000000], isReadFromLogEnd: [false], error: [None], [email-events-sox-verified,11] -> Fetch Data: [FetchDataInfo(1149863378 [1149653464 : 125581681],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 2, crc = 3823559139, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1262 cap=1262]),1149863378), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 4139901328, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1224 cap=1224]),1149863379), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 362718848, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1271 cap=1271]),1149863380), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 3513645936, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1265 cap=1265]),1149863381)))], HW: [1149863327], readSize: [50000000], isReadFromLogEnd: [false], error: [None], [email-events-sox-verified,3] -> Fetch Data: [FetchDataInfo(1149891362 [1149636809 : 160852840],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 2, crc = 712083310, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1244 cap=1244]),1149891362), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 1049476577, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1206 cap=1206]),1149891363), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 2736911333, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1199 cap=1199]),1149891364), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 1641412411, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1301 cap=1301]),1149891365), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 2563832660, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1299 cap=1299]),1149891366), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 54053881, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1269 cap=1269]),1149891367), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 3979896138, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1287 cap=1287]),1149891368)))], HW: [1149891315], readSize: [50000000], isReadFromLogEnd: [false], error: [None]) (kafka.server.ReplicaManager)
      [2016-07-28 13:31:36,711] ERROR [Replica Manager on Broker 157]: Error processing append operation on partition [contacts-pub-sub,7] (kafka.server.ReplicaManager)
      kafka.common.NotEnoughReplicasException: Number of insync replicas for partition [contacts-pub-sub,7] is [1], below required minimum [2]
      [2016-07-28 13:31:36,716] ERROR [Replica Manager on Broker 157]: Error processing append operation on partition [contacts-pub-sub,7] (kafka.server.ReplicaManager)
      kafka.common.NotEnoughReplicasException: Number of insync replicas for partition [contacts-pub-sub,7] is [1], below required minimum [2]
      [2016-07-28 13:31:36,727] DEBUG [Replica Manager on Broker 157]: Recording follower broker 160 log read results: Map([biden-rss-events,10] -> Fetch Data: [FetchDataInfo(2772927 [2692556 : 27087325],ByteBufferMessageSet())], HW: [2772927], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [cps-es-indexer-a,28] -> Fetch Data: [FetchDataInfo(519301 [496185 : 68076836],ByteBufferMessageSet())], HW: [519301], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [billing-amendment-applied-notifications,7] -> Fetch Data: [FetchDataInfo(2557 [2533 : 2856],ByteBufferMessageSet())], HW: [2557], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [usage-events-raw,1] -> Fetch Data: [FetchDataInfo(316637970 [316090773 : 831272496],ByteBufferMessageSet())], HW: [316637965], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [contacts-pub-sub,7] -> Fetch Data: [FetchDataInfo(2835905130 [2834198288 : 966360310],ByteBufferMessageSet())], HW: [2835905130], readSize: [50000000], isReadFromLogEnd: [true], error: [None], [email-events-sox-verified,3] -> Fetch Data: [FetchDataInfo(1149891363 [1149636809 : 160854110],FileMessageSet(MessageAndOffset(Message(magic = 0, attributes = 2, crc = 1049476577, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1206 cap=1206]),1149891363), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 2736911333, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1199 cap=1199]),1149891364), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 1641412411, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1301 cap=1301]),1149891365), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 2563832660, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1299 cap=1299]),1149891366), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 54053881, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1269 cap=1269]),1149891367), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 3979896138, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1287 cap=1287]),1149891368), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 3636950945, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1303 cap=1303]),1149891369), MessageAndOffset(Message(magic = 0, attributes = 2, crc = 3954674519, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=1210 cap=1210]),1149891370)))], HW: [1149891315], readSize: [50000000], isReadFromLogEnd: [false], error: [None]) (kafka.server.ReplicaManager)
      [2016-07-28 13:31:36,735] DEBUG Partition [contacts-pub-sub,7] on broker 157: Skipping update high watermark since Old hw 2835905130 [2834198288 : 966360310] is larger than new hw 2835905130 [2834198288 : 966360310] for partition [contacts-pub-sub,7]. All leo's are 2835905130 [2834198288 : 966360310] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,736] ERROR [Replica Manager on Broker 157]: Error processing append operation on partition [contacts-pub-sub,7] (kafka.server.ReplicaManager)
      kafka.common.NotEnoughReplicasException: Number of insync replicas for partition [contacts-pub-sub,7] is [1], below required minimum [2]
      [2016-07-28 13:31:36,736] DEBUG Partition [contacts-pub-sub,7] on broker 157: Recorded replica 158 log end offset (LEO) position 2835905122 for partition [contacts-pub-sub,7]. (kafka.cluster.Partition)
      [2016-07-28 13:31:36,740] INFO Partition [contacts-pub-sub,7] on broker 157: Expanding ISR for partition [contacts-pub-sub,7] from 157 to 157,160 (kafka.cluster.Partition)
      [2016-07-28 13:31:36,746] DEBUG Partition [contacts-pub-sub,7] on broker 157: Skipping update high watermark since Old hw 2835905130 [2834198288 : 966360310] is larger than new hw 2835905130 [2834198288 : 966360310] for partition [contacts-pub-sub,7]. All leo's are 2835905130 [2834198288 : 966360310] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,746] DEBUG Partition [contacts-pub-sub,7] on broker 157: Recorded replica 160 log end offset (LEO) position 2835905130 for partition [contacts-pub-sub,7]. (kafka.cluster.Partition)
      [2016-07-28 13:31:36,747] DEBUG Partition [contacts-pub-sub,7] on broker 157: Skipping update high watermark since Old hw 2835905130 [2834198288 : 966360310] is larger than new hw 2835905130 [2834198288 : 966360310] for partition [contacts-pub-sub,7]. All leo's are 2835905131 [2834198288 : 966360747],2835905130 [2834198288 : 966360310] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,750] DEBUG Partition [contacts-pub-sub,7] on broker 157: Skipping update high watermark since Old hw 2835905130 [2834198288 : 966360310] is larger than new hw 2835905130 [2834198288 : 966360310] for partition [contacts-pub-sub,7]. All leo's are 2835905132 [2834198288 : 966361180],2835905130 [2834198288 : 966360310] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,755] DEBUG Partition [contacts-pub-sub,7] on broker 157: Skipping update high watermark since Old hw 2835905130 [2834198288 : 966360310] is larger than new hw 2835905130 [2834198288 : 966360310] for partition [contacts-pub-sub,7]. All leo's are 2835905133 [2834198288 : 966361794],2835905130 [2834198288 : 966360310] (kafka.cluster.Partition)
      [2016-07-28 13:31:36,756] DEBUG Partition [contacts-pub-sub,7] on broker 157: Skipping update high watermark since Old hw 2835905130 [2834198288 : 966360310] is larger than new hw 2835905130 [2834198288 : 966360310] for partition [contacts-pub-sub,7]. All leo's are 2835905135 [2834198288 : 966363065],2835905130 [2834198288 : 966360310] (kafka.cluster.Partition)

      Attachments

        Activity

          People

            Unassigned Unassigned
            spackler Eric Abbott
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: