[KAFKA-4910] kafka consumer not receiving messages

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 0.10.0.0
• Component/s: consumer
• Assignee: Unassigned
• Reporter: Zhiwei Peng
• Created: 16/Mar/17 14:07
• Updated: 12/Sep/18 09:23
• Resolved: 12/Sep/18 09:23

Description

kafka consumer not receiving messages

consumer log：
"2017-03-10 14:35:34,448" | INFO | [Thread-5-KafkaSpout] | Revoking previously assigned partitions [MAFS_BPIS_ICSWIPE_IC-0] for group data_storm_hw_tianlu | org.apache.kafka.clients.consumer.internals.ConsumerCoordinator (ConsumerCoordinator.java:291)
"2017-03-10 14:35:34,451" | INFO | [Thread-5-KafkaSpout] | (Re-)joining group data_storm_hw_tianlu | org.apache.kafka.clients.consumer.internals.AbstractCoordinator (AbstractCoordinator.java:326)
"2017-03-10 14:35:34,456" | INFO | [Thread-5-KafkaSpout] | Successfully joined group data_storm_hw_tianlu with generation 7 | org.apache.kafka.clients.consumer.internals.AbstractCoordinator (AbstractCoordinator.java:434)
"2017-03-10 14:35:34,456" | INFO | [Thread-5-KafkaSpout] | Setting newly assigned partitions [MAFS_BPIS_ICSWIPE_IC-0] for group data_storm_hw_tianlu | org.apache.kafka.clients.consumer.internals.ConsumerCoordinator (ConsumerCoordinator.java:230)
"2017-03-10 14:36:10,297" | INFO | [Thread-7-__system] | Getting metrics for server on port 29104 | backtype.storm.messaging.netty.Server (Server.java:321)
"2017-03-10 14:36:10,298" | INFO | [Thread-7-__system] | Getting metrics for client connection to Netty-Client-streaming96/10.55.45.96:29106 | backtype.storm.messaging.netty.Client (Client.java:405)
"2017-03-10 14:37:10,297" | INFO | [Thread-7-__system] | Getting metrics for server on port 29104 | backtype.storm.messaging.netty.Server (Server.java:321)
"2017-03-10 14:37:10,298" | INFO | [Thread-7-__system] | Getting metrics for client connection to Netty-Client-streaming96/10.55.45.96:29106 | backtype.storm.messaging.netty.Client (Client.java:405)
"2017-03-10 14:38:10,297" | INFO | [Thread-7-__system] | Getting metrics for server on port 29104 | backtype.storm.messaging.netty.Server (Server.java:321)
"2017-03-10 14:38:10,297" | INFO | [Thread-7-__system] | Getting metrics for client connection to Netty-Client-streaming96/10.55.45.96:29106 | backtype.storm.messaging.netty.Client (Client.java:405)
"2017-03-10 14:39:10,297" | INFO | [Thread-7-__system] | Getting metrics for server on port 29104 | backtype.storm.messaging.netty.Server (Server.java:321)
"2017-03-10 14:39:10,298" | INFO | [Thread-7-__system] | Getting metrics for client connection to Netty-Client-streaming96/10.55.45.96:29106 | backtype.storm.messaging.netty.Client (Client.java:405)
"2017-03-10 14:40:10,298" | INFO | [Thread-7-__system] | Getting metrics for server on port 29104 | backtype.storm.messaging.netty.Server (Server.java:321)
"2017-03-10 14:40:10,298" | INFO | [Thread-7-__system] | Getting metrics for client connection to Netty-Client-streaming96/10.55.45.96:29106 | backtype.storm.messaging.netty.Client (Client.java:405)
"2017-03-10 14:40:34,454" | INFO | [Thread-5-KafkaSpout] | Revoking previously assigned partitions [MAFS_BPIS_ICSWIPE_IC-0] for group data_storm_hw_tianlu | org.apache.kafka.clients.consumer.internals.ConsumerCoordinator (ConsumerCoordinator.java:291)
"2017-03-10 14:40:34,457" | INFO | [Thread-5-KafkaSpout] | (Re-)joining group data_storm_hw_tianlu | org.apache.kafka.clients.consumer.internals.AbstractCoordinator (AbstractCoordinator.java:326)
"2017-03-10 14:40:36,458" | INFO | [Thread-5-KafkaSpout] | Successfully joined group data_storm_hw_tianlu with generation 8 | org.apache.kafka.clients.consumer.internals.AbstractCoordinator (AbstractCoordinator.java:434)
"2017-03-10 14:40:36,459" | INFO | [Thread-5-KafkaSpout] | Setting newly assigned partitions [MAFS_BPIS_ICSWIPE_IC-0] for group data_storm_hw_tianlu | org.apache.kafka.clients.consumer.internals.ConsumerCoordinator (ConsumerCoordinator.java:230)
"2017-03-10 14:41:10,297" | INFO | [Thread-7-__system] | Getting metrics for server on port 29104 | backtype.storm.messaging.netty.Server (Server.java:321)
"2017-03-10 14:41:10,298" | INFO | [Thread-7-__system] | Getting metrics for client connection to Netty-Client-streaming96/10.55.45.96:29106 | backtype.storm.messaging.netty.Client (Client.java:405)
"2017-03-10 14:42:10,297" | INFO | [Thread-7-__system] | Getting metrics for server on port 29104 | backtype.storm.messaging.netty.Server (Server.java:321)
"2017-03-10 14:42:10,298" | INFO | [Thread-7-__system] | Getting metrics for client connection to Netty-Client-streaming96/10.55.45.96:29106 | backtype.storm.messaging.netty.Client (Client.java:405)
"2017-03-10 14:43:10,297" | INFO | [Thread-7-__system] | Getting metrics for server on port 29104 | backtype.storm.messaging.netty.Server (Server.java:321)

kafka server log：
2017-03-10 14:39:40,767 | INFO | [kafka-scheduler-6] | Deleting index /srv/BigData/kafka/data5/kafka-logs/BDP_RS_LINENETWORKSPEED-0/00000000000008558684.index.deleted | kafka.log.OffsetIndex (Logging.scala:68)
2017-03-10 14:39:40,767 | INFO | [kafka-scheduler-0] | Deleting index /srv/BigData/kafka/data2/kafka-logs/BDP_RS_BUSLINESPEED-0/00000000000000000000.index.deleted | kafka.log.OffsetIndex (Logging.scala:68)
2017-03-10 14:40:34,457 | INFO | [kafka-request-handler-6] | [GroupCoordinator 2]: Preparing to restabilize group data_storm_hw_tianlu with old generation 7 | kafka.coordinator.GroupCoordinator (Logging.scala:68)
2017-03-10 14:40:36,454 | INFO | [kafka-request-handler-5] | [GroupCoordinator 2]: Stabilized group data_storm_hw_tianlu generation 8 | kafka.coordinator.GroupCoordinator (Logging.scala:68)
2017-03-10 14:40:36,455 | INFO | [kafka-request-handler-5] | [GroupCoordinator 2]: Assignment received from leader for group data_storm_hw_tianlu for generation 8 | kafka.coordinator.GroupCoordinator (Logging.scala:68)
2017-03-10 14:41:09,459 | INFO | [executor-Heartbeat] | [GroupCoordinator 2]: Preparing to restabilize group data_storm_hw_tianlu with old generation 8 | kafka.coordinator.GroupCoordinator (Logging.scala:68)
2017-03-10 14:41:12,460 | INFO | [kafka-request-handler-5] | [GroupCoordinator 2]: Stabilized group data_storm_hw_tianlu generation 9 | kafka.coordinator.GroupCoordinator (Logging.scala:68)
2017-03-10 14:41:12,462 | INFO | [kafka-request-handler-6] | [GroupCoordinator 2]: Assignment received from leader for group data_storm_hw_tianlu for generation 9 | kafka.coordinator.GroupCoordinator (Logging.scala:68)
2017-03-10 14:41:35,758 | INFO | [kafka-request-handler-0] | Rolled new log segment for 'MAFS_JSON_FC-0' in 1 ms. | kafka.log.Log (Logging.scala:68)
2017-03-10 14:42:19,540 | INFO | [kafka-request-handler-4] | Rolled new log segment for '__consumer_offsets-47' in 2 ms. | kafka.log.Log (Logging.scala:68)