[AVRO-2799] [Java] protobuf map types are not supported

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.9.2
• Component/s: java
• Assignee: Unassigned
• Reporter: Allen Wang
• Created: 13/Apr/20 17:26
• Updated: 12/Jun/20 08:20

Description

I have a protobuf message defined as this:

 

syntax = "proto3";

package ExampleProtobuf;

option java_package = "com.example";

import "google/protobuf/any.proto";
import "google/protobuf/timestamp.proto";
import "google/protobuf/wrappers.proto";

enum EventType {
  TYPE_UNKNOWN = 0;
  TYPE_ORDER_RECEIVED = 1;
  TYPE_PAYMENT_AUTHORIZED = 2;
  TYPE_PAYMENT_CAPTURED = 3;
  TYPE_ORDER_SUBMITTED = 4;
  TYPE_ORDER_DELIVERY_CREATED = 5;
  TYPE_MERCHANT_CONFIRM_ORDER = 6;
}

message RevenueEvent {
  google.protobuf.StringValue id = 1;
  EventType type = 2;
  google.protobuf.Timestamp change_time = 3;
  google.protobuf.StringValue country = 4;
  map <string, google.protobuf.Any> payload = 5;
  google.protobuf.StringValue source = 6;
  bool is_test = 7;
  google.protobuf.StringValue version = 8;
  google.protobuf.StringValue checksum = 9;
}

 

 

I wrote a program to encode such Protobuf message with Avro schema and convert it back to Protobuf message:

 

 

public class EventTest {
  public static void main(String[] args) throws  Exception {
    Example.RevenueEvent event = RevenueEvent.newBuilder()
            .setId(StringValue.newBuilder()
                    .setValue("12345")
                    .build())
            .setChangeTime(Timestamp.newBuilder()
                    .setSeconds(System.currentTimeMillis())
                    .build())
            .setChecksum(StringValue.newBuilder()
                    .setValue("xyz")
                    .build())
            .setCountry(StringValue.newBuilder()
                    .setValue("US")
                    .build())
            .setSource(StringValue.newBuilder()
                    .setValue("unknown")
                    .build())
            .setType(EventType.TYPE_MERCHANT_CONFIRM_ORDER)
            .putPayload("key", Any.newBuilder().setValue(ByteString.copyFromUtf8("xyz")).build())
            .build();
    ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
    Encoder encoder = EncoderFactory.get().directBinaryEncoder(outputStream, null);
    GenericDatumWriter<Example.RevenueEvent> datumWriter = new ProtobufDatumWriter<Example.RevenueEvent>(RevenueEvent.class);
    datumWriter.write(event, encoder);
    encoder.flush();
    byte[] bytes = outputStream.toByteArray();

    DatumReader<RevenueEvent> datumReader = new ProtobufDatumReader<>(RevenueEvent.class);
    InputStream inputStream = new ByteArrayInputStream(bytes);
    Decoder decoder = DecoderFactory.get().binaryDecoder(inputStream, null);
    datumReader.read(null, decoder);
  }
}

 

 

The program crashed with java.lang.StackOverflowError:

 

Exception in thread "main" java.lang.StackOverflowErrorException in thread "main" java.lang.StackOverflowError at java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) at org.apache.avro.specific.SpecificData.getClass(SpecificData.java:250) at org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:141) at org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143) at org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143) at org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143)

...

 

Looking into ProtoData.java, there seems to be a situation where the recursion would not end:

 

@Override
public Object newRecord(Object old, Schema schema) {
  try {
    Class c = SpecificData.get().getClass(schema);
    if (c == null)
      return newRecord(old, schema); // punt to generic <-- endless recursion
    if (c.isInstance(old))
      return old; // reuse instance
    return c.getMethod("newBuilder").invoke(null);

  } catch (Exception e) {
    throw new RuntimeException(e);
  }
}

Further experiment shows that if the map payload is not set, the decoding would be successful. 

 

protobuf-java:3.8.0 is used.

 

Issue Links

• is related to: AVRO-2780 ProtobufData and ThriftData Can Get Into Endless Loop