Avro
  1. Avro
  2. AVRO-1058

invalid int encoding with binary format

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.6.2, 1.6.3, 1.7.0
    • Fix Version/s: None
    • Component/s: java
    • Labels:
      None
    • Release Note:
      The same problems exist in the current 1.7.0-snapshot trunk. I didn't test versions prior to 1.6.2

      Description

      The java binary format sometimes generates an "invalid int encoding" exception and fails to roundtrip a record even though the json format roundtrips the same record just fine.

      In addition, there is a separate bug in that both binary and JSON format sometimes lead to an infinite loop when read() always returns null and never throws EOFException to indicate end-of-stream. This causes an OutOfMemoryError in the test driver because it forever adds null to a list of records.

      The attached test case java file demonstrates the problems. It walks all *.avsc and *.avpr files in the code base, generates random records based on those schemas, roundtrips the records, and then compares records pre and post roundtrip. To see it fail comment out portions of the following snippet:

      if (roundtripType == RoundtripType.BINARY_AVRO && schemaFile.getName().equals("weather.avsc") && i >= 350)

      { continue; // FIXME tmp work-around for avro bug (invalid int encoding on large string) }
      if (roundtripType == RoundtripType.BINARY_AVRO && schemaFile.getName().equals("Json.avsc") && i >= 1) { continue; // FIXME tmp work-around for avro bug (invalid int encoding on large string) }

      if (roundtripType == RoundtripType.BINARY_AVRO && schemaFile.getName().equals("WordCount.avsc") && i >= 2)

      { continue; // FIXME tmp work-around for avro bug (invalid int encoding on large string) }
      if (roundtripType == RoundtripType.BINARY_AVRO && schemaFile.getName().equals("mr_events.avpr") && i >= 0) { continue; // FIXME tmp work-around for avro bug (invalid int encoding on large string) }

      if (schemaFile.getName().equals("OnTheClasspath.avsc"))

      { continue; // FIXME tmp work-around for avro bug (OutOfMemoryError) }
      if (schemaFile.getName().equals("OnTheClasspath.avpr")) { continue; // FIXME tmp work-around for avro bug (OutOfMemoryError) }

      if (schemaFile.getName().equals("import.avpr"))

      { continue; // FIXME tmp work-around for avro bug (OutOfMemoryError) }
      if (schemaFile.getName().equals("namespaces.avpr")) { continue; // FIXME tmp work-around for avro bug (OutOfMemoryError) }

      Finally, there is a third separate issue, which is described in the javadoc for test method fixup():

      /**

      • You can trigger Record.equals() failures by modifying RandomData to spit
      • out Strings rather than Utf8 objects.
      • This hack replaces all occurances of Utf8 objects with String objects in
      • the given avro record tree. This is sometimes necessary to make
      • Record.equals() work correctly because Avro deserialization deserializes
      • String objects as Utf8 objects, and String.equals(Utf8) returns false
      • even if Utf8.equals(String) would return true.
      • In this particular test scenario this fixup hack might not be necessary
      • because the RandomData class always generates Utf8 instead of Strings.
      • Nonetheless, perhaps Record.equals() and descendants including Map
      • equality, etc, should treat any two pairs of String and Utf8 as equal if
      • string.equals(utf8.toString())). Perhaps Avro internals should arrange to
      • have the utf8 object always on the left hand side of equality
      • comparisons, like utf8.equals(obj).
        */
        private void fixup(Object obj) { ... }

      To summarize, there are really three separate issues here. I'm submitting them all in one bug report. Feel free to open separate JIRA issues if that's deemed more appropriate.

      1. TestRandomRecord.java
        13 kB
        wolfgang hoschek

        Activity

        Hide
        Thiruvalluvan M. G. added a comment -

        I've created a separate JIRA AVRO-1097 for the first problem.

        The second issue is because of certain record schemas that have no fields. Since Avro does not write anything for empty records, there is nothing to read. So even if the stream is at its end, no EOFException is thrown. In avro there is no way to distinguish streams with zero empty records from those having non-zero empty records. For empty records, such information should come from "out-of-band". I don't think it is a bug.

        The third problem seem to have been addressed already. All strings and char-sequences are converted into Utf8 for comparison in GenericData.compare().

        Show
        Thiruvalluvan M. G. added a comment - I've created a separate JIRA AVRO-1097 for the first problem. The second issue is because of certain record schemas that have no fields. Since Avro does not write anything for empty records, there is nothing to read. So even if the stream is at its end, no EOFException is thrown. In avro there is no way to distinguish streams with zero empty records from those having non-zero empty records. For empty records, such information should come from "out-of-band". I don't think it is a bug. The third problem seem to have been addressed already. All strings and char-sequences are converted into Utf8 for comparison in GenericData.compare().
        Hide
        wolfgang hoschek added a comment -

        test case code that demonstrates the issues

        Show
        wolfgang hoschek added a comment - test case code that demonstrates the issues

          People

          • Assignee:
            Unassigned
            Reporter:
            wolfgang hoschek
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development