[FLINK-22202] Thread safety in ParquetColumnarRowInputFormat

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.13.0
• Component/s: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
• Labels: pull-request-available
• Assignee: Jingsong Lee
• Reporter: Jingsong Lee
• Created: 12/Apr/21 02:11
• Updated: 07/Apr/22 06:32
• Resolved: 13/Apr/21 07:39

Description

In a VectorizedColumnBatch, the dictionary will be lazied deserialized. 

If there are multiple batches at the same time, there may be thread safety problems, because the deserialization of the dictionary depends on some internal structures.

We need set numBatchesToCirculate to 1 for ParquetColumnarRowInputFormat.

Issue Links

• duplicates: FLINK-21397 BufferUnderflowException when read parquet
• fixes: FLINK-20951 IllegalArgumentException when reading Hive parquet table if condition not contain all partitioned fields
• links to: GitHub Pull Request #15572