[FLINK-26116] FLIP-188: Implement Table Store V0.1

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: table-store-0.1.0
• Component/s: Table Store
• Assignee: Jingsong Lee
• Reporter: Jingsong Lee
• Created: 14/Feb/22 08:27
• Updated: 14/Apr/22 07:14
• Resolved: 14/Apr/22 07:14

Description

Introduce Table Store first version: v0.1

More detail see: https://cwiki.apache.org/confluence/display/FLINK/FLIP-188%3A+Introduce+Built-in+Dynamic+Table+Storage

Issue Links

• is related to: FLINK-25152 FLIP-188: Introduce Built-in Dynamic Table Storage
• relates to: FLINK-26653 [umbrella] Enhance Table Store SQL ITCases
• relates to: FLINK-26743 [umbrella] Build Document for Table Store

Sub-Tasks

• Init flink-table-store repository: Closed
• Introduce FileFormat for table-store: Closed
• Add basic structures of file store in table-store: Closed
• Introduce RecordReader and related classes for table store: Closed
• Introduce CompactStrategy and CompactManager for table store: Closed
• Introduce MergeTree writer and reader for table store: Closed
• Introduce Predicate to table store: Closed
• Introduce interfaces between file-table-store and flink connector sink: Closed
• Introduce Table Store Flink Sink: Closed
• Introduce atomic commit: Closed
• Implement partition and bucket filter of FileStoreScanImpl: Closed
• Introduce Table Store Flink Source: Closed
• Implement overwrite in FileStoreScanImpl: Closed
• Implement FileStoreExpire: Closed
• Support projection pushdown on keys and values in sst file readers: Closed
• Introduce FileStoreRead: Closed
• Introduce log store: Closed
• Migrate StoreSink to Sink V2: Closed
• Introduce implementation of ManagedTableFactory: Closed
• Introduce manifest.merge-min-count in commit: Closed
• Introduce a better expire strategy: Closed
• Introduce ITCase for table store source and sink: Closed
• Store log offsets in snapshot: Closed
• Add statistics collecting to sst files: Closed
• Integrate log store to StoreSink: Closed
• Overwrite commit kind should not be APPEND: Closed
• Allow ManifestFile to write a list of ManifestFileMeta into multiple files if too many.: Closed
• Rename Accumulator to MergeFunction: Closed
• FileStoreScan supports incremental consuming: Closed
• Record hasNull and allNull instead of nullCount in FieldStats: Closed
• Introduce TableStore API and refactor ITCases: Closed
• Introduce StoreTableSource and StoreTableSink: Closed
• Apply key & value filter in FileStoreScanImpl: Closed
• Introduce table.path option for FileStoreOptions: Closed
• FileStoreSourceSplitReader should deal with value count: Closed
• Delegate table planner dependencies with separate modules: Closed
• Introduce flink-table-store-dist module for packaging: Closed
• Remove change tracking in TableStore: Closed
• PK constraint should include partition key: Closed
• Introduce snapshot latest and earliest hint files: Closed
• Finish projection push down for table store: Closed
• Introduce auto type inference for filter push down: Closed
• Introduce parallelism setter for table store: Closed
• UniversalCompaction should pick by size ratio after picking by file num: Closed
• Use '<format>.<option>' instead of 'file.<format>.<option>' in table properties of managed table: Closed
• Reuse FileFormat instead of DecodingFormat/EncodingFormat to ensure thread safety.: Closed
• Introduce snapshot.num-retained.min option: Closed
• Introduce filter predicate for 'LIKE': Closed
• Rename 'file.path' table option in table store to 'path': Closed
• Disable rolling file when writing level 0 sst files: Closed
• FileStoreCommitImpl#filterCommitted should exit when faced with expired files: Closed
• Add version field to manifest entries: Closed
• Adjust default value of compaction and compress: Closed
• Add end to end tests for table store: Closed
• Prevent users from changing bucket number: Closed