[ARROW-11301] [C++] Fix reading LZ4-compressed Parquet files produced by Java Parquet implementation

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/27201
• Assignee: Antoine Pitrou
• Reporter: Antoine Pitrou
• Created: 18/Jan/21 12:53
• Updated: 11/Jan/23 08:18
• Resolved: 18/Jan/21 20:40

Description

We slightly misunderstood the Hadoop LZ4 format. A compressed buffer can actually contain several "frames", each prefixed with (de)compressed size.

See https://issues.apache.org/jira/browse/ARROW-9177?focusedCommentId=17267058&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17267058

Issue Links

• is related to: ARROW-9177 [C++][Parquet] Tracking issue for cross-implementation LZ4 Parquet compression compatibility
• relates to: PARQUET-1878 [C++] lz4 codec is not compatible with Hadoop Lz4Codec
• links to: GitHub Pull Request #9244