[PARQUET-2473] Clarify parquet-format with respect to repeated fields across boundaries

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: parquet-site
• Assignee: Andrew Lamb
• Reporter: Andrew Lamb
• Created: 15/May/24 14:30
• Updated: 23/Jun/24 03:33
• Resolved: 31/May/24 12:55

Description

Several implementors have reported that the parquet spec is currently unclear as to when repeated fields can span page boundaries (aka can a logical record be split across a page and/or row group boundary)

 

Discussion on list: https://lists.apache.org/thread/rd8twnvg4bg3558r507rzpxckcxt5wdn

 

The conclusion seems to be that the records can't be split across boundaries for "v2 data pages" or if there is a page index. 

 

We should clarify the spec to make this clear

Issue Links

• links to: GitHub Pull Request #244