[FLINK-33759] flink parquet writer support write nested array or map type

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.19.0
• Fix Version/s: 1.20.0
• Component/s: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Cai Liuyang
• Created: 06/Dec/23 05:24
• Updated: 12/Jun/24 20:46
• Resolved: 27/May/24 22:08

Description

When we use flink-parquet format wirte Map<String, String>[] type (which will be read by spark job), we encounter an exception:

// code placeholder
Caused by: org.apache.parquet.io.ParquetEncodingException: empty fields are illegal, the field should be ommited completely instead
    at org.apache.parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:329)
    at org.apache.flink.formats.parquet.row.ParquetRowDataWriter$ArrayWriter.writeArrayData(ParquetRowDataWriter.java:438)
    at org.apache.flink.formats.parquet.row.ParquetRowDataWriter$ArrayWriter.write(ParquetRowDataWriter.java:419)
    at org.apache.flink.formats.parquet.row.ParquetRowDataWriter$RowWriter.write(ParquetRowDataWriter.java:471)
    at org.apache.flink.formats.parquet.row.ParquetRowDataWriter.write(ParquetRowDataWriter.java:81)
    at org.apache.flink.formats.parquet.row.ParquetRowDataBuilder$ParquetWriteSupport.write(ParquetRowDataBuilder.java:89)

after review the code, we found flink-parquet doesn't support write nested array or map, because [ArrayWriter and MapWriter doesn't impl `public void write(ArrayData arrayData, int ordinal) {}` function.

Issue Links

• is related to: FLINK-17782 Add array,map,row types support for parquet row writer
• links to: GitHub Pull Request #23881
• links to: GitHub Pull Request #24029
• links to: GitHub Pull Request #24795