Uploaded image for project: 'Apache Avro'
  1. Apache Avro
  2. AVRO-2490

Unable to add field to existing avro schema

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 1.9.0, 1.8.2
    • Component/s: None
    • Labels:
      None

      Description

      As described in 

      https://docs.oracle.com/database/nosql-12.1.3.4/GettingStartedGuide/schemaevolution.html#changeschema-rules

       

      I should be able to extend AVRO schema by adding optional and(or?) default-having field. If I do that though, deserialization fails for me. It happens for version 1.8.2 and 1.9.0 as well.

       

      Some cases:

      A) If I add new field defined as:

      { "name": "newColumn", "type": ["null","string"], "default": null}

      deserialization will fail with:

       

      Caused by: java.lang.ArrayIndexOutOfBoundsException: 5 at org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:424) at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290) at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267) at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232) at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222) at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232) at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222) at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145) at tech.allegro.schema.json2avro.converter.JsonAvroConverter.convertToJson(JsonAvroConverter.java:83)

       

      if I add new field defined as:

      { "name": "newColumn", "type": "string", "default": ""}

      or

      { "name": "newColumn", "type": "string", "default": "incorrect"}

      (assuming I will fix it later in app)

      deserialization will fail with:

       

      Caused by: org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -1 at org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:336) at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:263) at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201) at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:422) at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:414) at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:181) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232) at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222) at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232) at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222) at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145) at tech.allegro.schema.json2avro.converter.JsonAvroConverter.convertToJson(JsonAvroConverter.java:83)

       

      what code does fail like that(that ± content of method convertToJson in JsonAvroConverter as seen in stacktrace.):

      BinaryDecoder binaryDecoder = DecoderFactory.get().binaryDecoder(avro, (BinaryDecoder)null);
      GenericRecord record = (GenericRecord)(new GenericDatumReader(schema)).read((Object)null, binaryDecoder);

      the second line produces given errors. Without new column added, deserialization works.

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              alfonz Martin Mucha
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: