[HUDI-7028] Fix Spark Quick Start

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.0.0
• Epic Link: filegroup-reader-writer-completion-time
• Assignee: Jonathan Vexler Assign to me
• Reporter: Lin Liu

Reviewers

Y Ethan Guo

• Created: 03/Nov/23 00:36
• Updated: 18/Jun/24 00:27

Description

Fix the bugs for Spark quick start when turning on file group reader and positional merging flag.

 

List some issues found so far:

1. [compatibility]When no positions are stored in the header, the read query failed. Idea behavior: use key based merging instead of failing.
2. [compatibility]When a parquet file contains Avro records, the file group reader of spark job will check if the payload is the expected type; otherwise, it will throw.