[PARQUET-1580] Page-level CRC checksum verification for DataPageV1

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.11.0
• Component/s: parquet-mr
• Labels: pull-request-available
• Assignee: Boudewijn Braams
• Reporter: Boudewijn Braams
• Created: 17/May/19 09:03
• Updated: 11/Mar/24 19:36
• Resolved: 24/Jul/19 06:35

Description

Now that the specification with respect to page level checksums has been clarified (see https://jira.apache.org/jira/browse/PARQUET-1539), we can actually implement it. What needs to be done:

• Implement calculating and writing checksums on write path
• Implement checksum validation on read path
• Make writing out checksums and validation on reads optional (default off)
• Implement tests

Issue Links

• causes: PARQUET-1746 Changed the data order after DataFrame reuse
• links to: GitHub Pull Request #647