[PARQUET-2446] ProtoParquetWriter Not Support DynamicMessage

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.1
• Fix Version/s: 1.14.0
• Component/s: parquet-protobuf
• Labels: Parquet parquetWriter

Environment

Parquet-Protobuf 1.13.1

• Assignee: fcyang
• Reporter: fcyang
• Created: 08/Mar/24 03:28
• Updated: 23/Jun/24 03:33
• Resolved: 15/Mar/24 01:56

Description

when i write DynamicMessages to parquet by protoParquetWriter,

pbwriter = ProtoParquetWriter.<DynamicMessage>builder(s3ParquetOutputFile)
        .withMessage(DynamicMessage.class)
        .withConf(hadoopConf)
        .withWriteMode(ParquetFileWriter.Mode.OVERWRITE)
        .withDictionaryEncoding(true)
        .withCompressionCodec(storage.conf().parquetCompressionCodecName())
        .withPageSize(PAGE_SIZE)
        .build(); 

I got an exception like this:

[ERROR] - com.twitter.elephantbird.util.Protobufs -Protobufs.java(126) -Could not find method getDescriptor in class com.google.protobuf.DynamicMessagecom.google.protobuf.DynamicMessage
java.lang.NoSuchMethodException: com.google.protobuf.DynamicMessage.getDescriptor()
        at java.lang.Class.getMethod(Class.java:1786)
        at com.twitter.elephantbird.util.Protobufs.getMessageDescriptor(Protobufs.java:123)
        at org.apache.parquet.proto.ProtoWriteSupport.init(ProtoWriteSupport.java:136)
        at org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:279)
        at org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:702) 

I believe that ProtoParquetWriter should support DynamicMessage and be capable of creating ProtoWriteSupport using descriptors  same with PBMessage class.

 

Issue Links

• links to: GitHub Pull Request #1295