Uploaded image for project: 'Avro'
  1. Avro
  2. AVRO-2137

avro JsonDecoding additional field in array type

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.8.1
    • Fix Version/s: None
    • Component/s: java
    • Labels:
      None

      Description

      I have following avro schema:

      {
        "type": "record",
        "name": "test",
        "namespace": "test.name",
        "fields": [
          {
            "name": "items",
            "type": {
              "type": "array",
              "items": {
                "type": "record",
                "name": "items",
                "fields": [
                  {
                    "name": "name",
                    "type": "string"
                  },
                  {
                    "name": "state",
                    "type": "string"
                  }
                ]
              }
            }
          },
          {
            "name": "firstname",
            "type": "string"
          }
        ]
      }
      

      when I am using Json decoder and avro encoder to encode Json data (scala code):

       

      
      val writer = new GenericDatumWriter[GenericRecord](schema)
      
      val reader = new GenericDatumReader[GenericRecord](schema)
      
      val baos = new ByteArrayOutputStream
      
      val decoder: JsonDecoder = DecoderFactory.get.jsonDecoder(schema, json)
      
      val encoder = EncoderFactory.get.binaryEncoder(baos, null)
      
      val datum = reader.read(null, decoder) writer.write(datum, encoder)
      
      encoder.flush()
      
      val avroByteArray = baos.toByteArray
      
       

      scenario1: when I am passing following json to encode it works fine:

       {
        "items": [
          {
            "name": "dallas",
            "state": "TX"
          }
        ],
        "firstname": "arun"
      }
      

       scenario2: when I am passing additional attribute in json at root level (lastname) it is able to encode and works fine:

      {
        "items": [
          {
            "name": "dallas",
            "state": "TX"
          }
        ],
        "firstname": "fname",
        "lastname": "lname"
      }
      

      scenario3: when I am add additional attribute in array record (country) it is throwing following exception:

      Expected record-end. Got FIELD_NAME org.apache.avro.AvroTypeException: Expected record-end. Got FIELD_NAME at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:698) { "items": [
      
      { "name": "dallas", "state": "TX", "country":"USA" }
      
      ], "firstname":"fname", "lastname":"lname" }
      

       In case of if we have any additional element in array type, it should work in same way as normal record; it should just discard them and decode the Json data.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              asethia Arun sethia
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: