[AVRO-1891] Generated Java code fails with union containing logical type

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.8.1
• Fix Version/s: 1.9.1
• Component/s: java, logical types
• Assignee: David Mollitor
• Reporter: Ross Black
• Created: 02/Aug/16 05:26
• Updated: 22/Jul/20 22:06
• Resolved: 01/Dec/19 10:58

Description

Example schema:

    {
      "type": "record",
      "name": "RecordV1",
      "namespace": "org.brasslock.event",
      "fields": [
        { "name": "first", "type": ["null", {"type": "long", "logicalType":"timestamp-millis"}]}
      ]
    }

The avro compiler generates a field using the relevant joda class:

    public org.joda.time.DateTime first

Running the following code to perform encoding:

        final RecordV1 record = new RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
        final DatumWriter<RecordV1> datumWriter = new SpecificDatumWriter<>(record.getSchema());
        final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
        final BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(stream, null);
        datumWriter.write(record, encoder);
        encoder.flush();
        final byte[] bytes = stream.toByteArray();

fails with the exception stacktrace:

 org.apache.avro.AvroRuntimeException: Unknown datum type org.joda.time.DateTime: 2016-07-29T10:15:30.000Z

    at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
    at org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
    at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
    at org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
    at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
    at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
    at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
    at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
    at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
    at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
    at org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

The failure can be fixed by explicitly adding the relevant conversion(s) to DatumWriter / SpecificData:

        final RecordV1 record = new RecordV1(DateTime.parse("2007-12-03T10:15:30.00Z"));
        final SpecificData specificData = new SpecificData();
        specificData.addLogicalTypeConversion(new TimeConversions.TimestampConversion());
        final DatumWriter<RecordV1> datumWriter = new SpecificDatumWriter<>(record.getSchema(), specificData);
        final ByteArrayOutputStream stream = new ByteArrayOutputStream(AvroUtil.DEFAULT_BUFFER_SIZE);
        final BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(stream, null);
        datumWriter.write(record, encoder);
        encoder.flush();
        final byte[] bytes = stream.toByteArray();

Issue Links

• blocks: AVRO-2163 Release 1.8.3
• relates to: AVRO-2065 Avro java code generation for Unions doesn't set converters for unions
• relates to: AVRO-2471 Java maven plugin code generation doesn't add conversion for timestamp-micros
• supercedes: AVRO-2212 GenericData.deepCopyRaw not working with java.lang.BigDecimal
• links to: GitHub Pull Request #118
• links to: GitHub Pull Request #118
• links to: GitHub Pull Request #329