Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-2431

Fix the checkpoint and change-log topic-creation workflow.

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      Symptom: Checkpoint and changelog kafka topics are expected to be created for a samza job with cleanup.policy set to compact. This ensures that all the data in the metadata topics are log-compacted. However, we observed that checkpoint & changelog topics for hundreds of production samza jobs were created with cleanup.policy set to delete leading to strong-possibility of message-l0ss.

      Cause:

      • Checkpoint: The control-flow in KafkaStreamSpec to build checkpoint spec swallows the essential kafka-topic configuration rather passes empty configuration bag to kafka-broker(via kafka-rest). This issue was introduced in SAMZA-2339.
      • Changelog: There change-log topic configurations are incorrectly generated when the RocksDB store TTL is set to -1 by the user.

      Changes: Change the topic-creation control-flow for above metadata topics to generate correct topic-configurations.

      Tests: Added unit tests to validate the expected topic configuration bag.

      API Changes: None

      Upgrade Instructions: None

      Usage Instructions: None

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            spvenkat Shanthoosh Venkataraman
            spvenkat Shanthoosh Venkataraman

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 2h 20m
                2h 20m

                Slack

                  Issue deployment