Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-1083

Do not load task stores which are older than delete tombstones during container startup

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Every local task store is backed up by a kafka changelog topic. Due to log compaction, delete tombstones of the changelog topic have a ttl of delete.retention.ms. Replaying the events from the changelog that has missing delete tombstones, would result in creation of an inconsistent local store(due to the missing of some delete events). This patch deletes the local stores in which difference between current time and last modified time of the offset file is greater than delete.retention.ms during the container startup.

      1. rb52476.patch
        20 kB
        Shanthoosh Venkataraman

        Activity

        Hide
        jmakes Jake Maes added a comment -

        Merged and committed. Thanks!

        Show
        jmakes Jake Maes added a comment - Merged and committed. Thanks!
        Hide
        spvenkat Shanthoosh Venkataraman added a comment -

        Jake Maes

        I've attached the updated patch in this ticket.

        ./bin/check-all.sh was successful with this patch.

        Thanks.

        Show
        spvenkat Shanthoosh Venkataraman added a comment - Jake Maes I've attached the updated patch in this ticket. ./bin/check-all.sh was successful with this patch. Thanks.
        Hide
        jmakes Jake Maes added a comment -

        Shanthoosh Venkataraman, no need to open a PR. I remember the old commit routine.

        I had a couple minor feedback items on the review. Address those and make sure this patch is rebased with the latest. Then I'll commit as soon as I can.

        Show
        jmakes Jake Maes added a comment - Shanthoosh Venkataraman , no need to open a PR. I remember the old commit routine. I had a couple minor feedback items on the review. Address those and make sure this patch is rebased with the latest. Then I'll commit as soon as I can.
        Hide
        spvenkat Shanthoosh Venkataraman added a comment -

        Jake Maes :

        I've attached the patch with this ticket.

        Here is the RB : https://reviews.apache.org/r/52476/ associated with this change. This will delete the task stores older than delete.retention.ms during the container startup.

        If a pull request has to be raised for this in github, I could do that as well.

        Thanks.

        Show
        spvenkat Shanthoosh Venkataraman added a comment - Jake Maes : I've attached the patch with this ticket. Here is the RB : https://reviews.apache.org/r/52476/ associated with this change. This will delete the task stores older than delete.retention.ms during the container startup. If a pull request has to be raised for this in github, I could do that as well. Thanks.

          People

          • Assignee:
            spvenkat Shanthoosh Venkataraman
            Reporter:
            spvenkat Shanthoosh Venkataraman
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development