Uploaded image for project: 'Apache Avro'
  1. Apache Avro
  2. AVRO-1661

Schema Evolution not working

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Invalid
    • 1.7.6, 1.7.7
    • None
    • java
    • Ubuntu 14.10

    Description

      This is the Avro Schema (OLD) I was using to write Avro binary data before:

      {
          "namespace": "com.hello.world",
          "type": "record",
          "name": "Toto",
          "fields": [
              {
                  "name": "a",
                  "type": [
                      "string",
                      "null"
                  ]
              },
              {
                  "name": "b",
                  "type": "string"
              }
          ]
      }
      

      This is the Avro Schema (NEW) I'm using to read the Avro binary data :

      {
          "namespace": "com.hello.world",
          "type": "record",
          "name": "Toto",
          "fields": [
              {
                  "name": "a",
                  "type": [
                      "string",
                      "null"
                  ]
              },
              {
                  "name": "b",
                  "type": "string"
              },
              {
                  "name": "c",
                  "type": "string",
                  "default": "na"
              }
          ]
      }
      

      However, I can't read the old data with the new Schema. I've got the following errors :

      15/04/08 17:32:22 ERROR executor.Executor: Exception in task 0.0 in stage 3.0 (TID 3)
      java.io.EOFException
      	at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:473)
      	at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:128)
      	at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:259)
      	at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:272)
      	at org.apache.avro.io.ValidatingDecoder.readString(ValidatingDecoder.java:113)
      	at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:353)
      	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:157)
      	at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
      	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
      	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
      	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
      	at com.miguno.kafka.avro.AvroDecoder.fromBytes(AvroDecoder.scala:31)
      

      From my understanding, I should be able to read the old data with the new schema that contains a new field with a default value. But it doesn't seem to work. Am I doing something wrong ?

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              nphung Nicolas PHUNG
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: