Minor Description
At HubSpot, we believe we hit a combination of bugs [1] [2], which may have caused us to lose data. In this scenario, as part of metadata conflict resolution a slowly starting up broker recovered an offset of zero and truncated segment files.
As part of a belt-and-suspenders approach to reducing this risk in the future, I propose adding the ability to rename/backup these files and allowing kafka to move on. Note that this breaks the ordering guarantees, but allows one to recover the data and decide later how to approach it.
This feature should be turned off by default but enabled with a configuration option.
(A pull request is following soon on Github)
1: https://issues.apache.org/jira/browse/KAFKA-2178
2: https://issues.apache.org/jira/browse/KAFKA-1120
Attachments
Issue Links
- links to