Avro
  1. Avro
  2. AVRO-1424

ValidatingDecoder hangs on large schema

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.5
    • Fix Version/s: 1.7.6
    • Component/s: c++
    • Labels:
      None

      Description

      Try to create Validation decoder using attached schema. It hangs and causes huge memory allocation. The problem is because of boost::any which is making excessive copies of objects when creating Symbols.

      And also fixup method which is being called in validation decoder creation stack is also taking long very very long time.

      1. model.avsc
        377 kB
        Ramana Suvarapu
      2. AVRO-1424.patch
        437 kB
        Thiruvalluvan M. G.

        Activity

        Hide
        Hudson added a comment -

        SUCCESS: Integrated in AvroJava #417 (See https://builds.apache.org/job/AvroJava/417/)
        AVRO-1424 ValidatingDecoder hangs on large schema (thiru: rev 1556285)

        • /avro/trunk/CHANGES.txt
        • /avro/trunk/lang/c++/CMakeLists.txt
        • /avro/trunk/lang/c++/impl/BinaryEncoder.cc
        • /avro/trunk/lang/c++/impl/Compiler.cc
        • /avro/trunk/lang/c++/impl/json/JsonDom.cc
        • /avro/trunk/lang/c++/impl/json/JsonDom.hh
        • /avro/trunk/lang/c++/impl/parsing/JsonCodec.cc
        • /avro/trunk/lang/c++/impl/parsing/ResolvingDecoder.cc
        • /avro/trunk/lang/c++/impl/parsing/Symbol.hh
        • /avro/trunk/lang/c++/impl/parsing/ValidatingCodec.cc
        • /avro/trunk/lang/c++/impl/parsing/ValidatingCodec.hh
        • /avro/trunk/lang/c++/jsonschemas/large_schema.avsc
        • /avro/trunk/lang/c++/test/JsonTests.cc
        • /avro/trunk/lang/c++/test/LargeSchemaTests.cc
        Show
        Hudson added a comment - SUCCESS: Integrated in AvroJava #417 (See https://builds.apache.org/job/AvroJava/417/ ) AVRO-1424 ValidatingDecoder hangs on large schema (thiru: rev 1556285) /avro/trunk/CHANGES.txt /avro/trunk/lang/c++/CMakeLists.txt /avro/trunk/lang/c++/impl/BinaryEncoder.cc /avro/trunk/lang/c++/impl/Compiler.cc /avro/trunk/lang/c++/impl/json/JsonDom.cc /avro/trunk/lang/c++/impl/json/JsonDom.hh /avro/trunk/lang/c++/impl/parsing/JsonCodec.cc /avro/trunk/lang/c++/impl/parsing/ResolvingDecoder.cc /avro/trunk/lang/c++/impl/parsing/Symbol.hh /avro/trunk/lang/c++/impl/parsing/ValidatingCodec.cc /avro/trunk/lang/c++/impl/parsing/ValidatingCodec.hh /avro/trunk/lang/c++/jsonschemas/large_schema.avsc /avro/trunk/lang/c++/test/JsonTests.cc /avro/trunk/lang/c++/test/LargeSchemaTests.cc
        Hide
        Thiruvalluvan M. G. added a comment -

        Committed revision 1556285. Thank you Ramana.

        Show
        Thiruvalluvan M. G. added a comment - Committed revision 1556285. Thank you Ramana.
        Hide
        ASF subversion and git services added a comment -

        Commit 1556285 from Thiruvalluvan M. G. in branch 'avro/trunk'
        [ https://svn.apache.org/r1556285 ]

        AVRO-1424 ValidatingDecoder hangs on large schema

        Show
        ASF subversion and git services added a comment - Commit 1556285 from Thiruvalluvan M. G. in branch 'avro/trunk' [ https://svn.apache.org/r1556285 ] AVRO-1424 ValidatingDecoder hangs on large schema
        Hide
        Ramana Suvarapu added a comment -

        Now ValidationDecoder is faster than the old one.

        Show
        Ramana Suvarapu added a comment - Now ValidationDecoder is faster than the old one.
        Hide
        Ramana Suvarapu added a comment -

        Hi Thiru, the patch worked and you can check-in the changes to trunk.

        Show
        Ramana Suvarapu added a comment - Hi Thiru, the patch worked and you can check-in the changes to trunk.
        Hide
        Thiruvalluvan M. G. added a comment -

        Yes Doug. I'm waiting for Ramana to confirm that this works for his real use cases. Of course, I've tested it with the sample schema he gave I just want to make sure that his other use cases are taken care, too.

        Show
        Thiruvalluvan M. G. added a comment - Yes Doug. I'm waiting for Ramana to confirm that this works for his real use cases. Of course, I've tested it with the sample schema he gave I just want to make sure that his other use cases are taken care, too.
        Hide
        Doug Cutting added a comment -

        This sounds like something we should perhaps include in 1.7.6, no?

        Show
        Doug Cutting added a comment - This sounds like something we should perhaps include in 1.7.6, no?
        Hide
        Thiruvalluvan M. G. added a comment -

        The patch is against the latest trunk. I tested again against a fresh checkout and it applies cleanly.

        Show
        Thiruvalluvan M. G. added a comment - The patch is against the latest trunk. I tested again against a fresh checkout and it applies cleanly.
        Hide
        Ramana Suvarapu added a comment -

        Unable to apply the patch as patch was not generated from Avro-1.7.5 version of source code.

        Show
        Ramana Suvarapu added a comment - Unable to apply the patch as patch was not generated from Avro-1.7.5 version of source code.
        Hide
        Thiruvalluvan M. G. added a comment -

        The attached patch addresses the problem.

        Show
        Thiruvalluvan M. G. added a comment - The attached patch addresses the problem.

          People

          • Assignee:
            Thiruvalluvan M. G.
            Reporter:
            Ramana Suvarapu
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development