Avro
  1. Avro
  2. AVRO-891

Change SpecificDatumReader to default reader schema from loaded class

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.4
    • Fix Version/s: 1.6.0
    • Component/s: java
    • Labels:
      None
    • Environment:

      OSX 10.7

      Description

      An AvroRuntimeException exception is thrown when attempting to read an Avro file serialized with an older version of a schema containing a field which has been subsequently removed in the newer schema.

      Exception

      Exception in thread "main" org.apache.avro.AvroRuntimeException: Bad index
      	at Record.put(Unknown Source)
      	at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
      	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
      	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
      	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
      	at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
      	at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220)
      	at Read.readFromAvro(Unknown Source)
      	at Read.main(Unknown Source)
      

      Steps to reproduce

      1. Generate code for schema v1 and v2
      2. Write an Avro file with the v1 code-generated Record class using the DataFileWriter and SpecificDatumWriter
      3. (informational only) Read the Avro file using the v1 code-generated Record class using DataFileStream and SpecificDatumReader (output follows)
        Record@2ec791b9[name=r1,id=1]
        Record@bd86fd3[name=r2,id=2]
        
      4. Read the Avro file using the v2 code-generated Record class using DataFileStream and SpecificDatumReader

      Schema details

      v1 schema:

      {"name": "Record", "type": "record",
        "fields": [
          {"name": "name", "type": "string"},
          {"name": "id", "type": "int"}
        ]
      }
      

      v2 schema:

      {"name": "Record", "type": "record",
        "fields": [
          {"name": "name", "type": "string"}
        ]
      }
      

      Write code

        public static Record createRecord(String name, int id) {
          Record record = new Record();
          record.name = name;
          record.id = id;
          return record;
        }
      
        public static void writeToAvro(OutputStream outputStream)
            throws IOException {
          DataFileWriter<Record> writer =
              new DataFileWriter<Record>(new SpecificDatumWriter<Record>());
          writer.create(Record.SCHEMA$, outputStream);
      
          writer.append(createRecord("r1", 1));
          writer.append(createRecord("r2", 2));
      
          writer.close();
          outputStream.close();
        }
      

      Read code

        public static void readFromAvro(InputStream is) throws IOException {
          DataFileStream<Record> reader = new DataFileStream<Record>(
                  is, new SpecificDatumReader<Record>());
          for (Record a : reader) {
            System.out.println(ToStringBuilder.reflectionToString(a));
          }
          IOUtils.cleanup(null, is);
          IOUtils.cleanup(null, reader);
        }
      
      
      1. AVRO-891.patch
        2 kB
        Doug Cutting
      2. AVRO-891.patch
        5 kB
        Doug Cutting

        Activity

        Doug Cutting made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Doug Cutting made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Doug Cutting made changes -
        Summary Change SpecificDatum Change SpecificDatumReader to default reader schema from loaded class
        Doug Cutting made changes -
        Summary Exception when attempting to read modified schema containing deleted field Change SpecificDatum
        Issue Type Bug [ 1 ] Improvement [ 4 ]
        Doug Cutting made changes -
        Attachment AVRO-891.patch [ 12496036 ]
        Doug Cutting made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Doug Cutting [ cutting ]
        Fix Version/s 1.6.0 [ 12316198 ]
        Doug Cutting made changes -
        Field Original Value New Value
        Attachment AVRO-891.patch [ 12495265 ]
        Alex Holmes created issue -

          People

          • Assignee:
            Doug Cutting
            Reporter:
            Alex Holmes
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development