Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-1083

Do not load task stores which are older than delete tombstones during container startup

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Every local task store is backed up by a kafka changelog topic. Due to log compaction, delete tombstones of the changelog topic have a ttl of delete.retention.ms. Replaying the events from the changelog that has missing delete tombstones, would result in creation of an inconsistent local store(due to the missing of some delete events). This patch deletes the local stores in which difference between current time and last modified time of the offset file is greater than delete.retention.ms during the container startup.

        Attachments

        1. rb52476.patch
          20 kB
          Shanthoosh Venkataraman

          Activity

            People

            • Assignee:
              spvenkat Shanthoosh Venkataraman
              Reporter:
              spvenkat Shanthoosh Venkataraman
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: