[BEAM-7925] ParquetIO supports neither column projection nor filter predicate

• Type: Improvement
• Status: Resolved
• Priority: P3
• Resolution: Won't Fix
• Affects Version/s: 2.14.0
• Fix Version/s: Missing
• Component/s: io-java-parquet
• Assignee: Unassigned
• Reporter: Neville Li
• Created: 07/Aug/19 21:16
• Updated: 13/Jan/22 21:52
• Resolved: 06/May/21 13:27

Description

Current ParquetIO supports neither column projection nor filter predicate which defeats the performance motivation of using Parquet in the first place. That's why we have our own implementation of ParquetIO in Scio.

Reading Parquet as Avro with column projection has some complications, namely, the resulting Avro records may be incomplete and will not survive ser/de. A workaround maybe provide a TypedRead interface that takes a Function<A, B> that maps invalid Avro A into user defined type B.

Issue Links

• blocks: BEAM-7929 ParquetTable.buildIOReader should support column projection and filter predicate
• links to: GitHub Pull Request #12786