[HUDI-7322] FileGroup Reader/Writer + Completion time semantics - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Add vote

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Epic
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: 1.0.0
Component/s: core
Labels:
None

Epic Name:
filegroup-reader-writer-completion-time

Description

This epic contains all the work done towards the new file group reader/writer interface along with new completion time semantics across all the query types.

Attachments

Sub-Tasks

Create Sub-Task

1.	Completion time incremental query for Spark reader	Open	Unassigned	Actions
2.	Completion time cdc query for Spark reader	Open	Unassigned	Actions
3.	Add IncrementalQueryAnalyzer for completion time based incremental queries	Open	Unassigned	Actions
4.	Support completion time inc queries with new FileGroupReader	Open	Unassigned	Actions

Issues in epic

quick-create-issue-for-epic-label

HUDI-7368	Integrate Flink with file group reader	Open	xy	Actions
HUDI-7543	Implement CDC query support (MoR/CoW) for Spark on FGReader	Open	Vinoth Chandar	Actions
HUDI-6787	Hive Integrate FileGroupReader with HoodieMergeOnReadSnapshotReader and RealtimeCompactedRecordReader for Hive	Closed	Jonathan Vexler	Actions
HUDI-7227	Enable completion time for File Group Reader	Open	Vinoth Chandar	Actions
HUDI-7075	Fix validation of parquet column projection on HadoopFsRelation in TestParquetColumnProjection	Open	Jonathan Vexler	Actions
HUDI-7157	Support filter pushdown for positional merging in Spark 3.5	Open	Lin Liu	Actions
HUDI-7269	Fallback to key-based merging if there is no positions in log header	Closed	Jonathan Vexler	Actions
HUDI-7065	Fix the new file group reader with COW in Spark integration	Open	Jonathan Vexler	Actions
HUDI-6798	Implement event-time-based merging mode in FileGroupReader	Closed	Ethan Guo	Actions
HUDI-7028	Fix Spark Quick Start	Open	Jonathan Vexler	Actions
HUDI-7599	Bootstrap MOR with legacy reader broken on master	Closed	Jonathan Vexler	Actions
HUDI-7605	Unable to set merger strategy with DataSourceWriteOptions.RECORD_MERGER_STRATEGY	Closed	Jonathan Vexler	Actions
HUDI-7607	Test with timestamp based key generator	Open	Jonathan Vexler	Actions