[KAFKA-9314] Connect put() and poll() retries not conforming to KIP-298

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: connect
• Assignee: Nigel Liang
• Reporter: Nigel Liang
• Created: 18/Dec/19 19:45
• Updated: 11/Jun/20 06:12
• Resolved: 11/Jun/20 06:12

Description

KIP-298 outlines the retry policy of Connect when errors are encountered. In particular, it proposes to retry on RetriableException on put() in SinkTask and poll() in SourceTask.

However, the code does not reflect this change. For instance, WorkerSourceTask handles RetriableException thrown from poll() by entering into a tight retry loop without backoff. This has led to connectors having to workaround by simply not retrying and failing the task always. Users would need to manually restart the task to recover from even simple network glitches.

AFAICT from reading code, the same is true for WorkerSinkTask when calling put().

Issue Links

• duplicates: KAFKA-9971 Error Reporting in Sink Connectors
• relates to: KAFKA-9971 Error Reporting in Sink Connectors
• links to: GitHub Pull Request #7857