Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3776

Unify store and downstream caching in streams

        Details

        • Type: New Feature
        • Status: Closed
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: 0.10.1.0
        • Fix Version/s: 0.10.1.0
        • Component/s: streams
        • Labels:
          None

          Description

          This is an umbrella story for capturing changes to processor caching in Streams as first described in KIP-63. https://cwiki.apache.org/confluence/display/KAFKA/KIP-63%3A+Unify+store+and+downstream+caching+in+streams

            Issue Links

            is duplicated by

            Improvement - An improvement or enhancement to an existing feature or task. KAFKA-3101 Optimize Aggregation Outputs

            • Major - Major loss of function.
            • Resolved
            is related to

            Improvement - An improvement or enhancement to an existing feature or task. KAFKA-3101 Optimize Aggregation Outputs

            • Major - Major loss of function.
            • Resolved
            links to

            Web Link GitHub Pull Request #1752

            mentioned in

            Page Loading...

            Page Loading...

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              GitHub user enothereska opened a pull request:

              https://github.com/apache/kafka/pull/1752

              KAFKA-3776: Unify store and downstream caching in streams [WiP]

              Work-in-progress PoC, not to be merged.

              You can merge this pull request into a Git repository by running:

              $ git pull https://github.com/enothereska/kafka KAFKA-3776-poc

              Alternatively you can review and apply these changes as the patch at:

              https://github.com/apache/kafka/pull/1752.patch

              To close this pull request, make a commit to your master/trunk branch
              with (at least) the following in the commit message:

              This closes #1752

              commit 9e6d1e0b7b50b8c9bb1848c9e72913701a93cbb0
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-16T09:45:00Z

              Initial commit with stub cache

              commit 2a3e46770b2aaddefd843f7cc6cc6727dd1cbc9e
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-17T08:22:23Z

              Adjustments so tests compile

              commit 86bb6dcb6beac3d136e01f45cd727e6a6a691b5b
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-17T10:21:11Z

              Remove old cache from RocksDbStore and add global cache to RocksDb

              Show
              githubbot ASF GitHub Bot added a comment - GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/1752 KAFKA-3776 : Unify store and downstream caching in streams [WiP] Work-in-progress PoC, not to be merged. You can merge this pull request into a Git repository by running: $ git pull https://github.com/enothereska/kafka KAFKA-3776 -poc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1752.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1752 commit 9e6d1e0b7b50b8c9bb1848c9e72913701a93cbb0 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-16T09:45:00Z Initial commit with stub cache commit 2a3e46770b2aaddefd843f7cc6cc6727dd1cbc9e Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-17T08:22:23Z Adjustments so tests compile commit 86bb6dcb6beac3d136e01f45cd727e6a6a691b5b Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-17T10:21:11Z Remove old cache from RocksDbStore and add global cache to RocksDb
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              Github user enothereska closed the pull request at:

              https://github.com/apache/kafka/pull/1752

              Show
              githubbot ASF GitHub Bot added a comment - Github user enothereska closed the pull request at: https://github.com/apache/kafka/pull/1752
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              GitHub user enothereska reopened a pull request:

              https://github.com/apache/kafka/pull/1752

              KAFKA-3776: Unify store and downstream caching in streams

              You can merge this pull request into a Git repository by running:

              $ git pull https://github.com/enothereska/kafka KAFKA-3776-poc

              Alternatively you can review and apply these changes as the patch at:

              https://github.com/apache/kafka/pull/1752.patch

              To close this pull request, make a commit to your master/trunk branch
              with (at least) the following in the commit message:

              This closes #1752

              commit 9e6d1e0b7b50b8c9bb1848c9e72913701a93cbb0
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-16T09:45:00Z

              Initial commit with stub cache

              commit 2a3e46770b2aaddefd843f7cc6cc6727dd1cbc9e
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-17T08:22:23Z

              Adjustments so tests compile

              commit 86bb6dcb6beac3d136e01f45cd727e6a6a691b5b
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-17T10:21:11Z

              Remove old cache from RocksDbStore and add global cache to RocksDb

              commit 989bce66ddbfc150081cbf74edc8d3d2b6ae70d3
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-17T11:27:08Z

              Create unique cache key per store

              commit 959885d4807e7f90030bca946dc5c1bdc4098754
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-17T12:51:17Z

              Enable caching for RocksDBWindowStore

              commit 6f582ceb98a5dbd37732e148a0e2f064a90d5d13
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-17T14:17:07Z

              Unit test for windowed stores

              commit 7810ab08b02b44e03812e9838062f811c64be0e7
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-19T14:45:27Z

              Plugged in new byte-based cache

              commit 333e7992b5f038d1d46d1a17d31a41b700552c77
              Author: Damian Guy <damian.guy@gmail.com>
              Date: 2016-08-22T15:07:00Z

              Add ProcessorRecordContext for tracking timestamp,offset etc of record. Update Procesor and Store APIs to use context. Dont forward values from Processors that are using a Store. RocksDBStores now have a flush listener and forward to processors whenever the cache is flushed. Left one failing test that highlights the need for the range queries to not forward values downstream

              commit 124ffdf84a42e0fe14f3be6897baecdb8f8b7ef0
              Author: Damian Guy <damian.guy@gmail.com>
              Date: 2016-08-22T15:56:36Z

              merge

              commit 34985fae49084b4ee37f0a1cc0c3fdd024a89056
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-23T06:58:49Z

              Initial pass at range queries. Cache based on TreeMap

              commit 30bde7301e7541cf2937f56b0303a43900db1fae
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-23T07:15:19Z

              Remove deleted entry from cache

              commit 9dbafc07701c5acd6d2c2edbb1f3770670180673
              Author: Damian Guy <damian.guy@gmail.com>
              Date: 2016-08-23T09:37:17Z

              revert api changes. Track RecordContext via ProcessorContext

              commit 63aacc465510bfc8cdb0b675b2d4ebb041a5e99c
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-23T13:18:19Z

              Pass at the 'all' method using cache iterator

              commit af6686fb298c2417c5eff9a46cbb8b898df894df
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-23T13:18:24Z

              Merge branch 'KAFKA-3776-poc' of https://github.com/enothereska/kafka into KAFKA-3776-poc

              commit 235b8612a121283841f8b3b8795ff6ff5c425c07
              Author: Damian Guy <damian.guy@gmail.com>
              Date: 2016-08-24T09:24:02Z

              disable caching for joins. expose enableCaching method on PersistentKeyValueFactory

              commit a6986a66b08d653b4aa019557f39f0713949fe83
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-24T10:17:30Z

              Cleanup memory cache

              commit 5b71d68d84007a03d59a3095467785da461d3c20
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-24T10:25:44Z

              Merged

              commit 7658e02a4b203d09d607c0a510d0ae3421eb71a9
              Author: Damian Guy <damian.guy@gmail.com>
              Date: 2016-08-24T11:21:35Z

              forward before changelog.

              commit 91bab64db1bf3673169879c80a15af67f4670b70
              Author: Damian Guy <damian.guy@gmail.com>
              Date: 2016-08-24T11:21:40Z

              Merge changes from Eno

              commit fef6cae098b44cf2e715db00c84d97650114802f
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-24T13:19:12Z

              Flush order should be top-down

              commit e1bcd35730a10c06a6e49d3882d787611b106f95
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-24T15:50:56Z

              Merged with trunk

              commit 591d4802383d6e8dc0a3e0e97cd619dd99d3db0e
              Author: Damian Guy <damian.guy@gmail.com>
              Date: 2016-08-24T19:35:28Z

              extract caching out of store

              commit dbd383c9066de031f48e8539ad071834b294b7b0
              Author: Damian Guy <damian.guy@gmail.com>
              Date: 2016-08-25T06:30:27Z

              Merge pull request #1 from enothereska/dg-3776-poc

              extract caching out of store

              commit 290a66b087847ceb67640574a7a6fc8c8e4d33db
              Author: Damian Guy <damian.guy@gmail.com>
              Date: 2016-08-25T07:12:55Z

              rename MergedSortedCacheRocksDBIterator -> MergedSortedCacheKeyValueStoreIterator

              commit 8a9981b16e59f2dcf656e698545a09733600a5b3
              Author: Damian Guy <damian.guy@gmail.com>
              Date: 2016-08-25T10:34:07Z

              refactor MergedSortedIterator. Some tests for MemoryLRUCacheIterator. Add PeekingKeyValueIterator

              commit c5f45775fb8ff275bd27bf6bef90290a53bb2199
              Author: Damian Guy <damian.guy@gmail.com>
              Date: 2016-08-25T14:37:21Z

              more tests. fix bugs

              commit 9ebdbd49164fdb281609bd8968c0d68453c2236c
              Author: Damian Guy <damian.guy@gmail.com>
              Date: 2016-08-25T16:54:53Z

              move all namespace related code into cache. make sure overlapping names do not clash

              commit 4b4228a27b5f0d5ab3f3153e7d2898423d230712
              Author: Damian Guy <damian.guy@gmail.com>
              Date: 2016-08-25T17:16:01Z

              add generic types to DelegatingPeekingKeyValueIterator and tests

              commit df7e77ac7264aa88dc9091f75ac93f217c38407c
              Author: Damian Guy <damian.guy@gmail.com>
              Date: 2016-08-26T10:20:18Z

              tidy up and tests

              commit f5cca5cedcbc11eb918c5f0c41474c228053e53d
              Author: Eno Thereska <eno.thereska@gmail.com>
              Date: 2016-08-26T10:25:02Z

              Isolate caches for each store

              Show
              githubbot ASF GitHub Bot added a comment - GitHub user enothereska reopened a pull request: https://github.com/apache/kafka/pull/1752 KAFKA-3776 : Unify store and downstream caching in streams You can merge this pull request into a Git repository by running: $ git pull https://github.com/enothereska/kafka KAFKA-3776 -poc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1752.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1752 commit 9e6d1e0b7b50b8c9bb1848c9e72913701a93cbb0 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-16T09:45:00Z Initial commit with stub cache commit 2a3e46770b2aaddefd843f7cc6cc6727dd1cbc9e Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-17T08:22:23Z Adjustments so tests compile commit 86bb6dcb6beac3d136e01f45cd727e6a6a691b5b Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-17T10:21:11Z Remove old cache from RocksDbStore and add global cache to RocksDb commit 989bce66ddbfc150081cbf74edc8d3d2b6ae70d3 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-17T11:27:08Z Create unique cache key per store commit 959885d4807e7f90030bca946dc5c1bdc4098754 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-17T12:51:17Z Enable caching for RocksDBWindowStore commit 6f582ceb98a5dbd37732e148a0e2f064a90d5d13 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-17T14:17:07Z Unit test for windowed stores commit 7810ab08b02b44e03812e9838062f811c64be0e7 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-19T14:45:27Z Plugged in new byte-based cache commit 333e7992b5f038d1d46d1a17d31a41b700552c77 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-22T15:07:00Z Add ProcessorRecordContext for tracking timestamp,offset etc of record. Update Procesor and Store APIs to use context. Dont forward values from Processors that are using a Store. RocksDBStores now have a flush listener and forward to processors whenever the cache is flushed. Left one failing test that highlights the need for the range queries to not forward values downstream commit 124ffdf84a42e0fe14f3be6897baecdb8f8b7ef0 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-22T15:56:36Z merge commit 34985fae49084b4ee37f0a1cc0c3fdd024a89056 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-23T06:58:49Z Initial pass at range queries. Cache based on TreeMap commit 30bde7301e7541cf2937f56b0303a43900db1fae Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-23T07:15:19Z Remove deleted entry from cache commit 9dbafc07701c5acd6d2c2edbb1f3770670180673 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-23T09:37:17Z revert api changes. Track RecordContext via ProcessorContext commit 63aacc465510bfc8cdb0b675b2d4ebb041a5e99c Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-23T13:18:19Z Pass at the 'all' method using cache iterator commit af6686fb298c2417c5eff9a46cbb8b898df894df Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-23T13:18:24Z Merge branch ' KAFKA-3776 -poc' of https://github.com/enothereska/kafka into KAFKA-3776 -poc commit 235b8612a121283841f8b3b8795ff6ff5c425c07 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-24T09:24:02Z disable caching for joins. expose enableCaching method on PersistentKeyValueFactory commit a6986a66b08d653b4aa019557f39f0713949fe83 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-24T10:17:30Z Cleanup memory cache commit 5b71d68d84007a03d59a3095467785da461d3c20 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-24T10:25:44Z Merged commit 7658e02a4b203d09d607c0a510d0ae3421eb71a9 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-24T11:21:35Z forward before changelog. commit 91bab64db1bf3673169879c80a15af67f4670b70 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-24T11:21:40Z Merge changes from Eno commit fef6cae098b44cf2e715db00c84d97650114802f Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-24T13:19:12Z Flush order should be top-down commit e1bcd35730a10c06a6e49d3882d787611b106f95 Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-24T15:50:56Z Merged with trunk commit 591d4802383d6e8dc0a3e0e97cd619dd99d3db0e Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-24T19:35:28Z extract caching out of store commit dbd383c9066de031f48e8539ad071834b294b7b0 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-25T06:30:27Z Merge pull request #1 from enothereska/dg-3776-poc extract caching out of store commit 290a66b087847ceb67640574a7a6fc8c8e4d33db Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-25T07:12:55Z rename MergedSortedCacheRocksDBIterator -> MergedSortedCacheKeyValueStoreIterator commit 8a9981b16e59f2dcf656e698545a09733600a5b3 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-25T10:34:07Z refactor MergedSortedIterator. Some tests for MemoryLRUCacheIterator. Add PeekingKeyValueIterator commit c5f45775fb8ff275bd27bf6bef90290a53bb2199 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-25T14:37:21Z more tests. fix bugs commit 9ebdbd49164fdb281609bd8968c0d68453c2236c Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-25T16:54:53Z move all namespace related code into cache. make sure overlapping names do not clash commit 4b4228a27b5f0d5ab3f3153e7d2898423d230712 Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-25T17:16:01Z add generic types to DelegatingPeekingKeyValueIterator and tests commit df7e77ac7264aa88dc9091f75ac93f217c38407c Author: Damian Guy <damian.guy@gmail.com> Date: 2016-08-26T10:20:18Z tidy up and tests commit f5cca5cedcbc11eb918c5f0c41474c228053e53d Author: Eno Thereska <eno.thereska@gmail.com> Date: 2016-08-26T10:25:02Z Isolate caches for each store
              Hide
              Permalink
              guozhang Guozhang Wang added a comment -

              Issue resolved by pull request 1752
              https://github.com/apache/kafka/pull/1752

              Show
              guozhang Guozhang Wang added a comment - Issue resolved by pull request 1752 https://github.com/apache/kafka/pull/1752
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              Github user asfgit closed the pull request at:

              https://github.com/apache/kafka/pull/1752

              Show
              githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1752

                People

                • Assignee:
                  enothereska Eno Thereska
                  Reporter:
                  enothereska Eno Thereska
                  Reviewer:
                  Eno Thereska
                • Votes:
                  1 Vote for this issue
                  Watchers:
                  7 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development