Avro
  1. Avro
  2. AVRO-1066

ArrayIndexOutOfBoundsException in ParsingEncoder when trying to use a json encoder to serialize a deep object graph

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.3
    • Fix Version/s: 1.7.0
    • Component/s: None
    • Labels:
      None

      Description

      I have an avro record that can have an array of children records of the same type. If this structure gets to be too deep then I continually get an ArrayOutOfBoundsException in ParsingEncoder when trying to use a json encoder. This works fine when using a binary encoder. My schema looks something like this:

      {
      "name" : "MyRecord",
      "type" : "record",
      "fields" : [

      { "name" : "fooField", "type" : "int" }

      ,
      {
      "name" : "childRecords",
      "type" : [ "null",

      { "type" : "array", "items" : "MyRecord" }

      ]
      }
      ]
      }

      The code I'm using to capture a JSON string for debugging looks like this:

      ByteArrayOutputStream os = null;
      try

      { os = new ByteArrayOutputStream(2048); final Encoder jsonEncoder = EncoderFactory.get().jsonEncoder(MyRecord.SCHEMA$, os); final DatumWriter<MyRecord> datumWriter = new SpecificDatumWriter<MyRecord>(MyRecord.class); datumWriter.write(record, jsonEncoder); jsonEncoder.flush(); return new String(os.toByteArray(), Charset.defaultCharset()); }

      catch (IOException e)

      { return null; }

      finally {
      if (os != null) {
      try

      { os.close(); }

      catch (IOException e) {
      }
      }
      }

      The error I get looks like this:
      java.lang.ArrayIndexOutOfBoundsException: 10
      at org.apache.avro.io.ParsingEncoder.push(ParsingEncoder.java:55)
      at org.apache.avro.io.JsonEncoder.writeArrayStart(JsonEncoder.java:231)
      at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:125)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
      at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
      at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
      at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
      at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
      at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
      at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
      at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
      at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
      at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
      at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
      at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
      at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
      at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
      at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
      at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
      at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
      at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
      at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
      at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
      at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
      at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
      at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
      <my line calling datumWriter.write>

      1. AVRO-1066.patch
        4 kB
        Thiruvalluvan M. G.
      2. AVRO-1066-2.patch
        1 kB
        Thiruvalluvan M. G.

        Activity

        Hide
        Daniel Lord added a comment -

        There was a post on the mailing list about this a few weeks ago. I just tested the suggested fix there and it works for me. Changing ParsingEncoder's push method to this worked for me.

        /** Push a new collection on to the stack. */
        protected final void push() {
        if (++pos >= counts.length)

        { counts = Arrays.copyOf(counts, pos + 10); }

        counts[pos] = 0;
        }

        Show
        Daniel Lord added a comment - There was a post on the mailing list about this a few weeks ago. I just tested the suggested fix there and it works for me. Changing ParsingEncoder's push method to this worked for me. /** Push a new collection on to the stack. */ protected final void push() { if (++pos >= counts.length) { counts = Arrays.copyOf(counts, pos + 10); } counts [pos] = 0; }
        Hide
        Thiruvalluvan M. G. added a comment -

        Good catch. Daniel Lord's fix would work.

        The problem has never been seen because we have not had nested array/maps with 10 levels or mores.

        Show
        Thiruvalluvan M. G. added a comment - Good catch. Daniel Lord's fix would work. The problem has never been seen because we have not had nested array/maps with 10 levels or mores.
        Hide
        Thiruvalluvan M. G. added a comment -

        Here is the patch with a unit test that demonstrates the problem.

        Show
        Thiruvalluvan M. G. added a comment - Here is the patch with a unit test that demonstrates the problem.
        Hide
        Thiruvalluvan M. G. added a comment -

        I'll commit this on Tuesday if there is no objection.

        Show
        Thiruvalluvan M. G. added a comment - I'll commit this on Tuesday if there is no objection.
        Hide
        Doug Cutting added a comment -

        Why the addition of the Accessor interface? A comment explaining this would probably be good to add too.

        Show
        Doug Cutting added a comment - Why the addition of the Accessor interface? A comment explaining this would probably be good to add too.
        Hide
        Thiruvalluvan M. G. added a comment -

        Oops! That was a mistake in the patch. I'll send a right patch.

        Show
        Thiruvalluvan M. G. added a comment - Oops! That was a mistake in the patch. I'll send a right patch.
        Hide
        Thiruvalluvan M. G. added a comment -

        Here is the correct patch.

        Show
        Thiruvalluvan M. G. added a comment - Here is the correct patch.
        Hide
        Doug Cutting added a comment -

        +1 The test fails for me without the patch and succeeds with it.

        Show
        Doug Cutting added a comment - +1 The test fails for me without the patch and succeeds with it.
        Hide
        Thiruvalluvan M. G. added a comment -

        Committed revision 1337322.

        Thank you Doug for reviewing it and I' sorry about the wrong initial patch.

        Show
        Thiruvalluvan M. G. added a comment - Committed revision 1337322. Thank you Doug for reviewing it and I' sorry about the wrong initial patch.

          People

          • Assignee:
            Daniel Lord
            Reporter:
            Daniel Lord
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development