[HUDI-5392] Fix Bootstrap files reader to configure arrays to be read in the new format - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Blocker
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.13.0
Component/s: bootstrap
Labels:
- pull-request-available

Story Points:
1
Epic Link:
Improve Bootstrap

Description

When writing Bootstrap file we’re using Spark writer that writes arrays in the new format, while Hudi reads it in the old (Avro compatible) format:

 // Old
 optional group tip_history (LIST) {
    repeated group array {
      optional double amount;
      optional binary currency (UTF8);
    }
  }

 // new
 optional group tip_history (LIST) {
    repeated group list {
      optional group element {
        optional double amount;
        optional binary currency (UTF8);
      }
    }
  }

To fix that we need to make sure that Bootstrap files are always read in a new format (Spark default) unlike Hudi's Parquet files

We also need to fix TestDataSourceForBootstrap, as it currently doesn't actually assert that the records are written correctly.