Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
We wanted to support reading 1.x tables in 0.16.0 release. So, creating this umbrella ticket to track all of them.
Changes required to be ported:
0. Creating 0.16.0 branch
0.a https://issues.apache.org/jira/browse/HUDI-7860 Completed.
1. Timeline
1.a Hoodie instant parsing should be able to read 1.x instants. https://issues.apache.org/jira/browse/HUDI-7883 Sagar.
1.b Commit metadata parsing is able to handle both json and avro formats. Scope might be non-trivial. https://issues.apache.org/jira/browse/HUDI-7866 Siva.
1.c HoodieDefaultTimeline able to read both timelines based on table version. https://issues.apache.org/jira/browse/HUDI-7884 Siva.
1.d Reading LSM timeline using 0.16.0 https://issues.apache.org/jira/browse/HUDI-7890 Siva.
1.e Ensure 1.0 MDT timeline is readable by 0.16 - HUDI-7901
2. Table property changes
2.a Table property changes https://issues.apache.org/jira/browse/HUDI-7885 https://issues.apache.org/jira/browse/HUDI-7865 LJ
3. MDT table changes
3.a record positions to RLI https://issues.apache.org/jira/browse/HUDI-7877 LJ
3.b MDT payload schema changes. https://issues.apache.org/jira/browse/HUDI-7886 LJ
4. Log format changes
4.a All metadata header types porting https://issues.apache.org/jira/browse/HUDI-7887 Jon
4.b Meaningful error for incompatible features from 1.x https://issues.apache.org/jira/browse/HUDI-7888 Jon
5. Log file slice or grouping detection compatibility
5. Tests
5.a Tests to validate that 1.x tables can be read w/ 0.16.0 https://issues.apache.org/jira/browse/HUDI-7896 Siva and Sagar.
6 Doc changes
6.a Call out unsupported features in 0.16.0 reader when reading 1.x tables. https://issues.apache.org/jira/browse/HUDI-7889