Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None

      Description

      For use cases like batch processing, samza need to re-process from beginning for each run. This also requires the ability to reset internal metadata streams so the processors will not accidentally read the meta data from previous runs, e.g. checkpoints. In order to support this use case, we need a new API in SystemAdmin to clear streams.

      For Kafka, clearStream will delete the topic in the brokers. This requires the following config for the brokers:

      delete.topic.enable=true

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user xinyuiscool opened a pull request:

          https://github.com/apache/samza/pull/292

          SAMZA-1415: Add clearStream API in SystemAdmin and remove deprecated APIs

          The patch does the following:

          1) add clearStream() APi in SystemAdmin. Currently it's only supported in Kafka with broker configuring delete.topic.enable=true.

          2) remove the deprecated APIs including createChangeLogStream(), validateChangelogStream() and createCoordinatorStream().

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/xinyuiscool/samza SAMZA-1415

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/samza/pull/292.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #292


          commit d2ae36f6074a1b224f43478e0b341837d41f061f
          Author: Xinyu Liu <xiliu@xiliu-ld1.linkedin.biz>
          Date: 2017-09-06T17:11:23Z

          Initial impl for clearStream

          commit 6b8298626c75b787058cc829de6fe012ed9f3875
          Author: Xinyu Liu <xiliu@xiliu-ld1.linkedin.biz>
          Date: 2017-09-06T23:29:31Z

          Clean up the deprecated api for creating coordinator and changelog streams


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user xinyuiscool opened a pull request: https://github.com/apache/samza/pull/292 SAMZA-1415 : Add clearStream API in SystemAdmin and remove deprecated APIs The patch does the following: 1) add clearStream() APi in SystemAdmin. Currently it's only supported in Kafka with broker configuring delete.topic.enable=true. 2) remove the deprecated APIs including createChangeLogStream(), validateChangelogStream() and createCoordinatorStream(). You can merge this pull request into a Git repository by running: $ git pull https://github.com/xinyuiscool/samza SAMZA-1415 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/samza/pull/292.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #292 commit d2ae36f6074a1b224f43478e0b341837d41f061f Author: Xinyu Liu <xiliu@xiliu-ld1.linkedin.biz> Date: 2017-09-06T17:11:23Z Initial impl for clearStream commit 6b8298626c75b787058cc829de6fe012ed9f3875 Author: Xinyu Liu <xiliu@xiliu-ld1.linkedin.biz> Date: 2017-09-06T23:29:31Z Clean up the deprecated api for creating coordinator and changelog streams

            People

            • Assignee:
              xinyu Xinyu Liu
              Reporter:
              xinyu Xinyu Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development