Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8395

Add an ability to backup log segment files on truncation

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: core
    • Labels:
    • Flags:
      Patch

      Description

      At HubSpot, we believe we hit a combination of bugs [1] [2], which may have caused us to lose data. In this scenario, as part of metadata conflict resolution a slowly starting up broker recovered an offset of zero and truncated segment files.

      As part of a belt-and-suspenders approach to reducing this risk in the future, I propose adding the ability to rename/backup these files and allowing kafka to move on. Note that this breaks the ordering guarantees, but allows one to recover the data and decide later how to approach it.

      This feature should be turned off by default but enabled with a configuration option.

      (A pull request is following soon on Github)

      1: https://issues.apache.org/jira/browse/KAFKA-2178
       2: https://issues.apache.org/jira/browse/KAFKA-1120

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                axiak Michael Axiak
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: