[KAFKA-10423] Logtash is restarting with invalid_fetch_session_epoch error

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.3.0
• Component/s: clients, connect, logging
• Assignee: Unassigned
• Reporter: Akshay Sharma
• Created: 20/Aug/20 09:49
• Updated: 25/Aug/20 10:14

Description

Logstash(input plugin is kafka) is restarting again and again with error as mentioned below

 

logs:

```

{"log":"[INFO ] 2020-07-27 05:03:43.873 [Ruby-0-Thread-12: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.3.1/lib/logstash/inputs/kafka.rb:244] FetchSessionHandler - [Consumer clientId=logstash-0, groupId=logstash] Node 1001 was unable to process the fetch request with (sessionId=2115239606, epoch=1128): INVALID_FETCH_SESSION_EPOCH.\n","stream":"stdout","time":"2020-07-27T05:03:43.873634303Z"}

{"log":"[WARN ] 2020-07-27 05:14:18.976 [SIGTERM handler] runner - SIGTERM received. Shutting down.\n","stream":"stdout","time":"2020-07-27T05:14:18.976808493Z"}

 

{"log":"[INFO ] 2020-07-27 05:14:19.030 [Ruby-0-Thread-12: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.3.1/lib/logstash/inputs/kafka.rb:244] AbstractCoordinator - [Consumer clientId=logstash-0, groupId=logstash] Sending LeaveGroup request to coordinator ikafka-0.kafka-headless.default.svc.cluster.local:9092 (id: 2147482646 rack: null)\n","stream":"stdout","time":"2020-07-27T05:14:19.031121275Z"}

```

KAFKA LOGS

```

{"log":"[2020-07-27 05:14:19,031] INFO [GroupCoordinator 1001]: Member logstash-0-9a59ad3d-c6ab-4dba-9775-5974d17934d1 in group logstash has left, removing it from the group (kafka.coordinator.group.GroupCoordinator)\n","stream":"stdout","time":"2020-07-27T05:14:19.032132241Z"}

{"log":"[2020-07-27 05:14:19,032] INFO [GroupCoordinator 1001]: Preparing to rebalance group logstash in state PreparingRebalance with old generation 85 (_consumer_offsets-49) (reason: removing member logstash-0-9a59ad3d-c6ab-4dba-9775-5974d17934d1 on LeaveGroup) (kafka.coordinator.group.GroupCoordinator)\n","stream":"stdout","time":"2020-07-27T05:14:19.032320407Z"} {"log":"[2020-07-27 05:14:19,032] INFO [GroupCoordinator 1001]: Group logstash with generation 86 is now empty (_consumer_offsets-49) (kafka.coordinator.group.GroupCoordinator)\n","stream":"stdout","time":"2020-07-27T05:14:19.032619661Z"}

 

{"log":"[2020-07-27 05:15:10,766] INFO [GroupCoordinator 1001]: Preparing to rebalance group logstash in state PreparingRebalance with old generation 86 (__consumer_offsets-49) (reason: Adding new member logstash-0-bde43584-a21e-4a0d-92cc-69196f213f11 with group instanceid None) (kafka.coordinator.group.GroupCoordinator)\n","stream":"stdout","time":"2020-07-27T05:15:10.767053896Z"}

```