Uploaded image for project: 'Apache Avro'
  1. Apache Avro
  2. AVRO-1891

Generated Java code fails with union containing logical type

    Details

    • Type: Bug
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: 1.8.1
    • Fix Version/s: 1.8.3
    • Component/s: java, logical types
    • Labels:
      None

      Description

      Example schema:

          {
            "type": "record",
            "name": "RecordV1",
            "namespace": "org.brasslock.event",
            "fields": [
              { "name": "first", "type": ["null", {"type": "long", "logicalType":"timestamp-millis"}]}
            ]
          }
      

      The avro compiler generates a field using the relevant joda class:

          public org.joda.time.DateTime first
      

      Running the following code to perform encoding:

              final RecordV1 record = new RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
              final DatumWriter<RecordV1> datumWriter = new SpecificDatumWriter<>(record.getSchema());
              final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
              final BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(stream, null);
              datumWriter.write(record, encoder);
              encoder.flush();
              final byte[] bytes = stream.toByteArray();
      

      fails with the exception stacktrace:

       org.apache.avro.AvroRuntimeException: Unknown datum type org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
      
          at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
          at org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
          at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
          at org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
          at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
          at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
          at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
          at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
          at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
          at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
          at org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
          at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
          at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
          at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
          at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
          at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
          at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
          at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
          at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
          at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
          at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
          at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
          at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
          at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
          at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
          at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
      

      The failure can be fixed by explicitly adding the relevant conversion(s) to DatumWriter / SpecificData:

              final RecordV1 record = new RecordV1(DateTime.parse("2007-12-03T10:15:30.00Z"));
              final SpecificData specificData = new SpecificData();
              specificData.addLogicalTypeConversion(new TimeConversions.TimestampConversion());
              final DatumWriter<RecordV1> datumWriter = new SpecificDatumWriter<>(record.getSchema(), specificData);
              final ByteArrayOutputStream stream = new ByteArrayOutputStream(AvroUtil.DEFAULT_BUFFER_SIZE);
              final BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(stream, null);
              datumWriter.write(record, encoder);
              encoder.flush();
              final byte[] bytes = stream.toByteArray();
      

        Attachments

        1. AVRO-1891.yshi.4.patch
          47 kB
          Yibing Shi
        2. AVRO-1891.yshi.3.patch
          50 kB
          Yibing Shi
        3. AVRO-1891.yshi.2.patch
          42 kB
          Yibing Shi
        4. AVRO-1891.yshi.1.patch
          35 kB
          Yibing Shi
        5. AVRO-1891.patch
          17 kB
          Tom White

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ross.black Ross Black
              • Votes:
                17 Vote for this issue
                Watchers:
                33 Start watching this issue

                Dates

                • Created:
                  Updated: