[AVRO-2137] avro JsonDecoding additional field in array type

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.8.1
• Component/s: java
• Assignee: Unassigned
• Reporter: Arun sethia
• Created: 29/Jan/18 02:31
• Updated: 01/Mar/19 15:21

Description

I have following avro schema:

{
  "type": "record",
  "name": "test",
  "namespace": "test.name",
  "fields": [
    {
      "name": "items",
      "type": {
        "type": "array",
        "items": {
          "type": "record",
          "name": "items",
          "fields": [
            {
              "name": "name",
              "type": "string"
            },
            {
              "name": "state",
              "type": "string"
            }
          ]
        }
      }
    },
    {
      "name": "firstname",
      "type": "string"
    }
  ]
}

when I am using Json decoder and avro encoder to encode Json data (scala code):

 

val writer = new GenericDatumWriter[GenericRecord](schema)

val reader = new GenericDatumReader[GenericRecord](schema)

val baos = new ByteArrayOutputStream

val decoder: JsonDecoder = DecoderFactory.get.jsonDecoder(schema, json)

val encoder = EncoderFactory.get.binaryEncoder(baos, null)

val datum = reader.read(null, decoder) writer.write(datum, encoder)

encoder.flush()

val avroByteArray = baos.toByteArray

 

scenario1: when I am passing following json to encode it works fine:

 {
  "items": [
    {
      "name": "dallas",
      "state": "TX"
    }
  ],
  "firstname": "arun"
}

 scenario2: when I am passing additional attribute in json at root level (lastname) it is able to encode and works fine:

{
  "items": [
    {
      "name": "dallas",
      "state": "TX"
    }
  ],
  "firstname": "fname",
  "lastname": "lname"
}

scenario3: when I am add additional attribute in array record (country) it is throwing following exception:

Expected record-end. Got FIELD_NAME org.apache.avro.AvroTypeException: Expected record-end. Got FIELD_NAME at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:698) { "items": [

{ "name": "dallas", "state": "TX", "country":"USA" }

], "firstname":"fname", "lastname":"lname" }

 In case of if we have any additional element in array type, it should work in same way as normal record; it should just discard them and decode the Json data.