[FLINK-7733] test instability in Kafka end-to-end test (RetriableCommitFailedException)

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Cannot Reproduce
• Affects Version/s: 1.4.0
• Component/s: Connectors / Kafka, Tests
• Labels: test-stability
• Assignee: Unassigned
• Reporter: Nico Kruber
• Created: 28/Sep/17 15:24
• Updated: 01/Mar/18 15:23
• Resolved: 01/Mar/18 15:23

Description

In a branch with unrelated changes, the Kafka end-to-end tests fails with the following strange entries in the log.

2017-09-27 17:50:36,777 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka version : 0.10.2.1
2017-09-27 17:50:36,778 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka commitId : e89bffd6b2eff799
2017-09-27 17:50:38,492 INFO  org.apache.kafka.clients.consumer.internals.AbstractCoordinator  - Discovered coordinator testing-gce-b6810926-7003-4390-9002-178739bb4946:9092 (id: 2147483647 rack: null) for group myconsumer.
2017-09-27 17:50:38,511 INFO  org.apache.kafka.clients.consumer.internals.AbstractCoordinator  - Marking the coordinator testing-gce-b6810926-7003-4390-9002-178739bb4946:9092 (id: 2147483647 rack: null) dead for group myconsumer
2017-09-27 17:50:38,618 INFO  org.apache.kafka.clients.consumer.internals.AbstractCoordinator  - Discovered coordinator testing-gce-b6810926-7003-4390-9002-178739bb4946:9092 (id: 2147483647 rack: null) for group myconsumer.
2017-09-27 17:50:41,525 INFO  org.apache.flink.runtime.state.DefaultOperatorStateBackend    - DefaultOperatorStateBackend snapshot (In-Memory Stream Factory, synchronous part) in thread Thread[Async calls on Source: Custom Source -> Map -> Sink: Unnamed (1/1),5,Flink Task Threads] took 481 ms.
2017-09-27 17:50:41,598 INFO  org.apache.kafka.clients.consumer.internals.AbstractCoordinator  - Marking the coordinator testing-gce-b6810926-7003-4390-9002-178739bb4946:9092 (id: 2147483647 rack: null) dead for group myconsumer
2017-09-27 17:50:41,600 WARN  org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher  - Committing offsets to Kafka failed. This does not compromise Flink's checkpoints.
org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset commit failed with a retriable exception. You should retry committing offsets.
Caused by: org.apache.kafka.common.errors.DisconnectException
2017-09-27 17:50:41,608 ERROR org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase  - Async Kafka commit failed.
org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset commit failed with a retriable exception. You should retry committing offsets.
Caused by: org.apache.kafka.common.errors.DisconnectException

https://travis-ci.org/NicoK/flink/jobs/280477275