[AVRO-891] Change SpecificDatumReader to default reader schema from loaded class

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.4
• Fix Version/s: 1.6.0
• Component/s: java

Environment

OSX 10.7

• Assignee: Doug Cutting
• Reporter: Alex Holmes
• Created: 20/Sep/11 10:42
• Updated: 01/Nov/11 18:36
• Resolved: 22/Sep/11 19:09

Description

An AvroRuntimeException exception is thrown when attempting to read an Avro file serialized with an older version of a schema containing a field which has been subsequently removed in the newer schema.

Exception

Exception in thread "main" org.apache.avro.AvroRuntimeException: Bad index
	at Record.put(Unknown Source)
	at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
	at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
	at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220)
	at Read.readFromAvro(Unknown Source)
	at Read.main(Unknown Source)

Steps to reproduce

1. Generate code for schema v1 and v2
2. Write an Avro file with the v1 code-generated Record class using the DataFileWriter and SpecificDatumWriter
3. (informational only) Read the Avro file using the v1 code-generated Record class using DataFileStream and SpecificDatumReader (output follows)

   Record@2ec791b9[name=r1,id=1]
   Record@bd86fd3[name=r2,id=2]
   

4. Read the Avro file using the v2 code-generated Record class using DataFileStream and SpecificDatumReader

Schema details

v1 schema:

{"name": "Record", "type": "record",
  "fields": [
    {"name": "name", "type": "string"},
    {"name": "id", "type": "int"}
  ]
}

v2 schema:

{"name": "Record", "type": "record",
  "fields": [
    {"name": "name", "type": "string"}
  ]
}

Write code

  public static Record createRecord(String name, int id) {
    Record record = new Record();
    record.name = name;
    record.id = id;
    return record;
  }

  public static void writeToAvro(OutputStream outputStream)
      throws IOException {
    DataFileWriter<Record> writer =
        new DataFileWriter<Record>(new SpecificDatumWriter<Record>());
    writer.create(Record.SCHEMA$, outputStream);

    writer.append(createRecord("r1", 1));
    writer.append(createRecord("r2", 2));

    writer.close();
    outputStream.close();
  }

Read code

  public static void readFromAvro(InputStream is) throws IOException {
    DataFileStream<Record> reader = new DataFileStream<Record>(
            is, new SpecificDatumReader<Record>());
    for (Record a : reader) {
      System.out.println(ToStringBuilder.reflectionToString(a));
    }
    IOUtils.cleanup(null, is);
    IOUtils.cleanup(null, reader);
  }