Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3633

AvroStorage tests are failing when running against Avro 1.7.5

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      AvroStorage tests executed against latest Avro release 1.7.5 are failing:

      ant clean test -Dtestcase=TestAvroStorage -Davro.version=1.7.5
      ...
         [junit] Running org.apache.pig.builtin.TestAvroStorage
         [junit] Tests run: 33, Failures: 15, Errors: 0, Time elapsed: 14.211 sec
      

      With following exception:

      Failed to parse: Pig script failed to parse: 
      <line 1, column 5> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'AvroStorage' with arguments '[, -r]'
      	at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
      	at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1676)
      	at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1623)
      	at org.apache.pig.PigServer.registerQuery(PigServer.java:575)
      	at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1093)
      	at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501)
      	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
      	at org.apache.pig.PigServer.registerScript(PigServer.java:649)
      	at org.apache.pig.PigServer.registerScript(PigServer.java:726)
      	at org.apache.pig.PigServer.registerScript(PigServer.java:699)
      	at org.apache.pig.builtin.TestAvroStorage.testAvroStorage(TestAvroStorage.java:775)
      	at org.apache.pig.builtin.TestAvroStorage.testLoadRecursiveRecordsOptionOn(TestAvroStorage.java:588)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:601)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      	at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
      	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
      	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
      	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
      Caused by: 
      <line 1, column 5> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'AvroStorage' with arguments '[, -r]'
      	at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:849)
      	at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3479)
      	at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1536)
      	at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1013)
      	at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:553)
      	at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
      	at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
      	... 34 more
      Caused by: java.lang.RuntimeException: could not instantiate 'AvroStorage' with arguments '[, -r]'
      	at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:748)
      	at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:837)
      	... 40 more
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
      	at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:716)
      	... 41 more
      Caused by: java.lang.NullPointerException
      	at org.apache.avro.Schema.parse(Schema.java:1072)
      	at org.apache.avro.Schema$Parser.parse(Schema.java:950)
      	at org.apache.avro.Schema$Parser.parse(Schema.java:940)
      	at org.apache.pig.builtin.AvroStorage.<init>(AvroStorage.java:136)
      	... 46 more
      3701 [main] WARN  org.apache.pig.tools.parameters.PreprocessorContext  - Warning : Multiple values found for INFILE. Using value test/org/apache/pig/builtin/avro/data/trevni/uncompressed/simpleRecordsTrevni.trevni
      3701 [main] WARN  org.apache.pig.tools.parameters.PreprocessorContext  - Warning : Multiple values found for AVROSTORAGE_OUT_2. Using value -f test/org/apache/pig/builtin/avro/schema/simpleRecordsTrevni.avsc
      3702 [main] WARN  org.apache.pig.tools.parameters.PreprocessorContext  - Warning : Multiple values found for OUTFILE. Using value /home/jarcec/cloudera/repos/pig/build/test/TestAvroStorage/testLoadTrevniRecords
      3732 [main] ERROR org.apache.pig.PigServer  - exception during parsing: Error during parsing. Pig script failed to parse: 
      <line 4, column 0> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'AvroStorage' with arguments '[, -f test/org/apache/pig/builtin/avro/schema/simpleRecordsTrevni.avsc]'
      Failed to parse: Pig script failed to parse: 
      <line 4, column 0> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'AvroStorage' with arguments '[, -f test/org/apache/pig/builtin/avro/schema/simpleRecordsTrevni.avsc]'
      	at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
      	at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1676)
      	at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1623)
      	at org.apache.pig.PigServer.registerQuery(PigServer.java:575)
      	at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1093)
      	at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501)
      	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
      	at org.apache.pig.PigServer.registerScript(PigServer.java:649)
      	at org.apache.pig.PigServer.registerScript(PigServer.java:726)
      	at org.apache.pig.PigServer.registerScript(PigServer.java:699)
      	at org.apache.pig.builtin.TestAvroStorage.testAvroStorage(TestAvroStorage.java:775)
      	at org.apache.pig.builtin.TestAvroStorage.testLoadTrevniRecords(TestAvroStorage.java:709)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:601)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      	at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
      	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
      	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
      	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
      Caused by: 
      <line 4, column 0> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'AvroStorage' with arguments '[, -f test/org/apache/pig/builtin/avro/schema/simpleRecordsTrevni.avsc]'
      	at org.apache.pig.parser.LogicalPlanBuilder.buildStoreOp(LogicalPlanBuilder.java:936)
      	at org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator.java:7691)
      	at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1580)
      	at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1013)
      	at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:553)
      	at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
      	at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
      	... 34 more
      Caused by: java.lang.RuntimeException: could not instantiate 'AvroStorage' with arguments '[, -f test/org/apache/pig/builtin/avro/schema/simpleRecordsTrevni.avsc]'
      	at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:748)
      	at org.apache.pig.parser.LogicalPlanBuilder.buildStoreOp(LogicalPlanBuilder.java:915)
      	... 40 more
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
      	at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:716)
      	... 41 more
      Caused by: java.lang.NullPointerException
      	at org.apache.avro.Schema.parse(Schema.java:1072)
      	at org.apache.avro.Schema$Parser.parse(Schema.java:950)
      	at org.apache.avro.Schema$Parser.parse(Schema.java:940)
      	at org.apache.pig.builtin.AvroStorage.<init>(AvroStorage.java:136)
      	... 46 more
      

      Thomas White has looked into the problem and found out that Avro in version 1.7.5 has updated the JSON parser library Jackson from 1.8.8 to 1.9.13. The new version is changing the way empty strings are processed.

        Attachments

        1. PIG-3633.patch
          0.5 kB
          Jarek Jarcec Cecho

          Activity

            People

            • Assignee:
              jarcec Jarek Jarcec Cecho
              Reporter:
              jarcec Jarek Jarcec Cecho
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: