[BEAM-13157] ParquetIO.parseGenericRecords does not support Configuration

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.35.0
• Component/s: io-java-parquet
• Assignee: Michael Luckey
• Reporter: Michael Luckey
• Created: 01/Nov/21 15:59
• Updated: 08/Nov/21 14:45
• Resolved: 03/Nov/21 19:41

Description

ParquetIO.Parse currently does not support a Hadoop configuration.

Although it is possible to set the configuration object, it is ignored later on. Parse delegates to ParseFiles, but misses to pass the configuration object here. Therefore, config is lost.

Workaround:
Use ParseFiles directly by unrolling Parse.expand implementation.

Issue Links

• links to: GitHub Pull Request #15856
• links to: GitHub Pull Request #15914