[KAFKA-3522] Consider adding version information into rocksDB storage format

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Component/s: streams
• Labels: architecture
• Assignee: Matthias J. Sax
• Reporter: Guozhang Wang
• Created: 06/Apr/16 23:08
• Updated: 17/May/19 22:50
• Resolved: 17/May/19 00:54

Description

Kafka Streams does not introduce any modifications to the data format in the underlying Kafka protocol, but it does use RocksDB for persistent state storage, and currently its data format is fixed and hard-coded. We want to consider the evolution path in the future we we change the data format, and hence having some version info stored along with the storage file / directory would be useful.

And this information could be even out of the storage file; for example, we can just use a small "version indicator" file in the rocksdb directory for this purposes. Thoughts? enothereska jkreps

Issue Links

• blocks: KAFKA-6521 Store record timestamps in KTable stores
• blocks: KAFKA-7869 Refactor RocksDBConfigSetter API to separate DBOptions and CFOptions
• contains: KAFKA-4304 Extend Interactive Queries for return latest update timestamp per key
• relates to: KAFKA-6054 ERROR "SubscriptionInfo - unable to decode subscription data: version=2" when upgrading from 0.10.0.0 to 0.10.2.1
• links to: Follow up PR
• links to: GitHub Pull Request #5422
• links to: GitHub Pull Request #6044
• links to: GitHub Pull Request #6149
• links to: GitHub Pull Request #6150
• links to: GitHub Pull Request #6151
• links to: GitHub Pull Request #6152
• links to: GitHub Pull Request #6170
• links to: GitHub Pull Request #6173
• links to: GitHub Pull Request #6175
• links to: GitHub Pull Request #6186
• links to: GitHub Pull Request #6204
• links to: GitHub Pull Request #6222
• links to: GitHub Pull Request #6356
• links to: GitHub Pull Request #6661
• links to: GitHub Pull Request #6756
• mentioned in: Page Loading...
• mentioned in: Page Loading...