Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3776

Unify store and downstream caching in streams

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.1.0
    • Fix Version/s: 0.10.1.0
    • Component/s: streams
    • Labels:
      None

      Description

      This is an umbrella story for capturing changes to processor caching in Streams as first described in KIP-63. https://cwiki.apache.org/confluence/display/KAFKA/KIP-63%3A+Unify+store+and+downstream+caching+in+streams

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/kafka/pull/1752

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1752
          Hide
          guozhang Guozhang Wang added a comment -

          Issue resolved by pull request 1752
          https://github.com/apache/kafka/pull/1752

          Show
          guozhang Guozhang Wang added a comment - Issue resolved by pull request 1752 https://github.com/apache/kafka/pull/1752
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user enothereska reopened a pull request:

          https://github.com/apache/kafka/pull/1752

          KAFKA-3776: Unify store and downstream caching in streams

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/enothereska/kafka KAFKA-3776-poc

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/kafka/pull/1752.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1752


          commit 9e6d1e0b7b50b8c9bb1848c9e72913701a93cbb0
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-16T09:45:00Z

          Initial commit with stub cache

          commit 2a3e46770b2aaddefd843f7cc6cc6727dd1cbc9e
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-17T08:22:23Z

          Adjustments so tests compile

          commit 86bb6dcb6beac3d136e01f45cd727e6a6a691b5b
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-17T10:21:11Z

          Remove old cache from RocksDbStore and add global cache to RocksDb

          commit 989bce66ddbfc150081cbf74edc8d3d2b6ae70d3
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-17T11:27:08Z

          Create unique cache key per store

          commit 959885d4807e7f90030bca946dc5c1bdc4098754
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-17T12:51:17Z

          Enable caching for RocksDBWindowStore

          commit 6f582ceb98a5dbd37732e148a0e2f064a90d5d13
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-17T14:17:07Z

          Unit test for windowed stores

          commit 7810ab08b02b44e03812e9838062f811c64be0e7
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-19T14:45:27Z

          Plugged in new byte-based cache

          commit 333e7992b5f038d1d46d1a17d31a41b700552c77
          Author: Damian Guy <damian.guy@gmail.com>
          Date: 2016-08-22T15:07:00Z

          Add ProcessorRecordContext for tracking timestamp,offset etc of record. Update Procesor and Store APIs to use context. Dont forward values from Processors that are using a Store. RocksDBStores now have a flush listener and forward to processors whenever the cache is flushed. Left one failing test that highlights the need for the range queries to not forward values downstream

          commit 124ffdf84a42e0fe14f3be6897baecdb8f8b7ef0
          Author: Damian Guy <damian.guy@gmail.com>
          Date: 2016-08-22T15:56:36Z

          merge

          commit 34985fae49084b4ee37f0a1cc0c3fdd024a89056
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-23T06:58:49Z

          Initial pass at range queries. Cache based on TreeMap

          commit 30bde7301e7541cf2937f56b0303a43900db1fae
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-23T07:15:19Z

          Remove deleted entry from cache

          commit 9dbafc07701c5acd6d2c2edbb1f3770670180673
          Author: Damian Guy <damian.guy@gmail.com>
          Date: 2016-08-23T09:37:17Z

          revert api changes. Track RecordContext via ProcessorContext

          commit 63aacc465510bfc8cdb0b675b2d4ebb041a5e99c
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-23T13:18:19Z

          Pass at the 'all' method using cache iterator

          commit af6686fb298c2417c5eff9a46cbb8b898df894df
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-23T13:18:24Z

          Merge branch 'KAFKA-3776-poc' of https://github.com/enothereska/kafka into KAFKA-3776-poc

          commit 235b8612a121283841f8b3b8795ff6ff5c425c07
          Author: Damian Guy <damian.guy@gmail.com>
          Date: 2016-08-24T09:24:02Z

          disable caching for joins. expose enableCaching method on PersistentKeyValueFactory

          commit a6986a66b08d653b4aa019557f39f0713949fe83
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-24T10:17:30Z

          Cleanup memory cache

          commit 5b71d68d84007a03d59a3095467785da461d3c20
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-24T10:25:44Z

          Merged

          commit 7658e02a4b203d09d607c0a510d0ae3421eb71a9
          Author: Damian Guy <damian.guy@gmail.com>
          Date: 2016-08-24T11:21:35Z

          forward before changelog.

          commit 91bab64db1bf3673169879c80a15af67f4670b70
          Author: Damian Guy <damian.guy@gmail.com>
          Date: 2016-08-24T11:21:40Z

          Merge changes from Eno

          commit fef6cae098b44cf2e715db00c84d97650114802f
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-24T13:19:12Z

          Flush order should be top-down

          commit e1bcd35730a10c06a6e49d3882d787611b106f95
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-24T15:50:56Z

          Merged with trunk

          commit 591d4802383d6e8dc0a3e0e97cd619dd99d3db0e
          Author: Damian Guy <damian.guy@gmail.com>
          Date: 2016-08-24T19:35:28Z

          extract caching out of store

          commit dbd383c9066de031f48e8539ad071834b294b7b0
          Author: Damian Guy <damian.guy@gmail.com>
          Date: 2016-08-25T06:30:27Z

          Merge pull request #1 from enothereska/dg-3776-poc

          extract caching out of store

          commit 290a66b087847ceb67640574a7a6fc8c8e4d33db
          Author: Damian Guy <damian.guy@gmail.com>
          Date: 2016-08-25T07:12:55Z

          rename MergedSortedCacheRocksDBIterator -> MergedSortedCacheKeyValueStoreIterator

          commit 8a9981b16e59f2dcf656e698545a09733600a5b3
          Author: Damian Guy <damian.guy@gmail.com>
          Date: 2016-08-25T10:34:07Z

          refactor MergedSortedIterator. Some tests for MemoryLRUCacheIterator. Add PeekingKeyValueIterator

          commit c5f45775fb8ff275bd27bf6bef90290a53bb2199
          Author: Damian Guy <damian.guy@gmail.com>
          Date: 2016-08-25T14:37:21Z

          more tests. fix bugs

          commit 9ebdbd49164fdb281609bd8968c0d68453c2236c
          Author: Damian Guy <damian.guy@gmail.com>
          Date: 2016-08-25T16:54:53Z

          move all namespace related code into cache. make sure overlapping names do not clash

          commit 4b4228a27b5f0d5ab3f3153e7d2898423d230712
          Author: Damian Guy <damian.guy@gmail.com>
          Date: 2016-08-25T17:16:01Z

          add generic types to DelegatingPeekingKeyValueIterator and tests

          commit df7e77ac7264aa88dc9091f75ac93f217c38407c
          Author: Damian Guy <damian.guy@gmail.com>
          Date: 2016-08-26T10:20:18Z

          tidy up and tests

          commit f5cca5cedcbc11eb918c5f0c41474c228053e53d
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-26T10:25:02Z

          Isolate caches for each store


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user enothereska reopened a pull request: https://github.com/apache/kafka/pull/1752 KAFKA-3776 : Unify store and downstream caching in streams You can merge this pull request into a Git repository by running: $ git pull https://github.com/enothereska/kafka KAFKA-3776 -poc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1752.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1752 commit 9e6d1e0b7b50b8c9bb1848c9e72913701a93cbb0 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-16T09:45:00Z Initial commit with stub cache commit 2a3e46770b2aaddefd843f7cc6cc6727dd1cbc9e Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-17T08:22:23Z Adjustments so tests compile commit 86bb6dcb6beac3d136e01f45cd727e6a6a691b5b Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-17T10:21:11Z Remove old cache from RocksDbStore and add global cache to RocksDb commit 989bce66ddbfc150081cbf74edc8d3d2b6ae70d3 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-17T11:27:08Z Create unique cache key per store commit 959885d4807e7f90030bca946dc5c1bdc4098754 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-17T12:51:17Z Enable caching for RocksDBWindowStore commit 6f582ceb98a5dbd37732e148a0e2f064a90d5d13 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-17T14:17:07Z Unit test for windowed stores commit 7810ab08b02b44e03812e9838062f811c64be0e7 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-19T14:45:27Z Plugged in new byte-based cache commit 333e7992b5f038d1d46d1a17d31a41b700552c77 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-22T15:07:00Z Add ProcessorRecordContext for tracking timestamp,offset etc of record. Update Procesor and Store APIs to use context. Dont forward values from Processors that are using a Store. RocksDBStores now have a flush listener and forward to processors whenever the cache is flushed. Left one failing test that highlights the need for the range queries to not forward values downstream commit 124ffdf84a42e0fe14f3be6897baecdb8f8b7ef0 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-22T15:56:36Z merge commit 34985fae49084b4ee37f0a1cc0c3fdd024a89056 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-23T06:58:49Z Initial pass at range queries. Cache based on TreeMap commit 30bde7301e7541cf2937f56b0303a43900db1fae Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-23T07:15:19Z Remove deleted entry from cache commit 9dbafc07701c5acd6d2c2edbb1f3770670180673 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-23T09:37:17Z revert api changes. Track RecordContext via ProcessorContext commit 63aacc465510bfc8cdb0b675b2d4ebb041a5e99c Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-23T13:18:19Z Pass at the 'all' method using cache iterator commit af6686fb298c2417c5eff9a46cbb8b898df894df Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-23T13:18:24Z Merge branch ' KAFKA-3776 -poc' of https://github.com/enothereska/kafka into KAFKA-3776 -poc commit 235b8612a121283841f8b3b8795ff6ff5c425c07 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-24T09:24:02Z disable caching for joins. expose enableCaching method on PersistentKeyValueFactory commit a6986a66b08d653b4aa019557f39f0713949fe83 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-24T10:17:30Z Cleanup memory cache commit 5b71d68d84007a03d59a3095467785da461d3c20 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-24T10:25:44Z Merged commit 7658e02a4b203d09d607c0a510d0ae3421eb71a9 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-24T11:21:35Z forward before changelog. commit 91bab64db1bf3673169879c80a15af67f4670b70 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-24T11:21:40Z Merge changes from Eno commit fef6cae098b44cf2e715db00c84d97650114802f Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-24T13:19:12Z Flush order should be top-down commit e1bcd35730a10c06a6e49d3882d787611b106f95 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-24T15:50:56Z Merged with trunk commit 591d4802383d6e8dc0a3e0e97cd619dd99d3db0e Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-24T19:35:28Z extract caching out of store commit dbd383c9066de031f48e8539ad071834b294b7b0 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-25T06:30:27Z Merge pull request #1 from enothereska/dg-3776-poc extract caching out of store commit 290a66b087847ceb67640574a7a6fc8c8e4d33db Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-25T07:12:55Z rename MergedSortedCacheRocksDBIterator -> MergedSortedCacheKeyValueStoreIterator commit 8a9981b16e59f2dcf656e698545a09733600a5b3 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-25T10:34:07Z refactor MergedSortedIterator. Some tests for MemoryLRUCacheIterator. Add PeekingKeyValueIterator commit c5f45775fb8ff275bd27bf6bef90290a53bb2199 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-25T14:37:21Z more tests. fix bugs commit 9ebdbd49164fdb281609bd8968c0d68453c2236c Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-25T16:54:53Z move all namespace related code into cache. make sure overlapping names do not clash commit 4b4228a27b5f0d5ab3f3153e7d2898423d230712 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-25T17:16:01Z add generic types to DelegatingPeekingKeyValueIterator and tests commit df7e77ac7264aa88dc9091f75ac93f217c38407c Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-26T10:20:18Z tidy up and tests commit f5cca5cedcbc11eb918c5f0c41474c228053e53d Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-26T10:25:02Z Isolate caches for each store
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user enothereska closed the pull request at:

          https://github.com/apache/kafka/pull/1752

          Show
          githubbot ASF GitHub Bot added a comment - Github user enothereska closed the pull request at: https://github.com/apache/kafka/pull/1752
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user enothereska opened a pull request:

          https://github.com/apache/kafka/pull/1752

          KAFKA-3776: Unify store and downstream caching in streams [WiP]

          Work-in-progress PoC, not to be merged.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/enothereska/kafka KAFKA-3776-poc

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/kafka/pull/1752.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1752


          commit 9e6d1e0b7b50b8c9bb1848c9e72913701a93cbb0
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-16T09:45:00Z

          Initial commit with stub cache

          commit 2a3e46770b2aaddefd843f7cc6cc6727dd1cbc9e
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-17T08:22:23Z

          Adjustments so tests compile

          commit 86bb6dcb6beac3d136e01f45cd727e6a6a691b5b
          Author: Eno Thereska <eno.thereska@gmail.com>
          Date: 2016-08-17T10:21:11Z

          Remove old cache from RocksDbStore and add global cache to RocksDb


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/1752 KAFKA-3776 : Unify store and downstream caching in streams [WiP] Work-in-progress PoC, not to be merged. You can merge this pull request into a Git repository by running: $ git pull https://github.com/enothereska/kafka KAFKA-3776 -poc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1752.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1752 commit 9e6d1e0b7b50b8c9bb1848c9e72913701a93cbb0 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-16T09:45:00Z Initial commit with stub cache commit 2a3e46770b2aaddefd843f7cc6cc6727dd1cbc9e Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-17T08:22:23Z Adjustments so tests compile commit 86bb6dcb6beac3d136e01f45cd727e6a6a691b5b Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-17T10:21:11Z Remove old cache from RocksDbStore and add global cache to RocksDb

            People

            • Assignee:
              enothereska Eno Thereska
              Reporter:
              enothereska Eno Thereska
              Reviewer:
              Eno Thereska
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development