Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-1694

Clash in protobuf versions between Avatica and Hadoop

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.12.0
    • Component/s: avatica, pig-adapter
    • Labels:
      None

      Description

      When running Pig adapter tests, we get the following stack:

      java.lang.NoClassDefFoundError: com/google/protobuf/GeneratedMessageV3
      
      	at java.lang.ClassLoader.defineClass1(Native Method)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
      	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
      	at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
      	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      	at org.apache.calcite.avatica.ConnectionPropertiesImpl.<clinit>(ConnectionPropertiesImpl.java:37)
      	at org.apache.calcite.avatica.MetaImpl.<init>(MetaImpl.java:71)
      	at org.apache.calcite.jdbc.CalciteMetaImpl.<init>(CalciteMetaImpl.java:82)
      	at org.apache.calcite.jdbc.Driver.createMeta(Driver.java:169)
      	at org.apache.calcite.avatica.AvaticaConnection.<init>(AvaticaConnection.java:118)
      	at org.apache.calcite.jdbc.CalciteConnectionImpl.<init>(CalciteConnectionImpl.java:113)
      	at org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection.<init>(CalciteJdbc41Factory.java:114)
      	at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:59)
      	at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:44)
      	at org.apache.calcite.jdbc.CalciteFactory.newConnection(CalciteFactory.java:53)
      	at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
      	at java.sql.DriverManager.getConnection(DriverManager.java:664)
      	at java.sql.DriverManager.getConnection(DriverManager.java:208)
      	at org.apache.calcite.test.CalciteAssert$MapConnectionFactory.createConnection(CalciteAssert.java:1151)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.createConnection(CalciteAssert.java:1189)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1260)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1350)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1345)
      	at org.apache.calcite.test.PigAdapterTest.testImplWithGroupByMultipleFields(PigAdapterTest.java:116)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:483)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
      	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
      	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
      	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
      	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:483)
      	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
      Caused by: java.lang.ClassNotFoundException: com.google.protobuf.GeneratedMessageV3
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      	... 58 more
      

      followed by

      java.lang.NoClassDefFoundError: Could not initialize class org.apache.calcite.avatica.ConnectionPropertiesImpl
      
      	at org.apache.calcite.avatica.MetaImpl.<init>(MetaImpl.java:71)
      	at org.apache.calcite.jdbc.CalciteMetaImpl.<init>(CalciteMetaImpl.java:82)
      	at org.apache.calcite.jdbc.Driver.createMeta(Driver.java:169)
      	at org.apache.calcite.avatica.AvaticaConnection.<init>(AvaticaConnection.java:118)
      	at org.apache.calcite.jdbc.CalciteConnectionImpl.<init>(CalciteConnectionImpl.java:113)
      	at org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection.<init>(CalciteJdbc41Factory.java:114)
      	at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:59)
      	at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:44)
      	at org.apache.calcite.jdbc.CalciteFactory.newConnection(CalciteFactory.java:53)
      	at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
      	at java.sql.DriverManager.getConnection(DriverManager.java:664)
      	at java.sql.DriverManager.getConnection(DriverManager.java:208)
      	at org.apache.calcite.test.CalciteAssert$MapConnectionFactory.createConnection(CalciteAssert.java:1151)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.createConnection(CalciteAssert.java:1189)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1260)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1350)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1345)
      	at org.apache.calcite.test.PigAdapterTest.testImplWithMultipleFilters(PigAdapterTest.java:59)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:483)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
      	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
      	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
      	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
      	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:483)
      	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
      

      If you explicitly load ConnectionPropertiesImpl by adding the line

      Util.discard(org.apache.calcite.avatica.ConnectionPropertiesImpl.class);
      

      to PigSchemaFactory the error goes away.

      I think the problem might be that the Pig adapter pulls in Pig, which pulls in Hadoop, which depends on protobuf 2, whereas Avatica's ConnectionPropertiesImpl has some members that depend on protobuf 3.

      We can work around by having the Pig adapter load Calcite's MetaImpl (thereby avoiding an explicit dependency of the Pig adapter on Avatica).

      But the real solution, I think, is too reduce the dependencies of ConnectionPropertiesImpl. Calcite (and other Avatica users) do not care about the protobuf implementation of Avatica's wire protocol, but protobuf gets loaded due to the fromProto(Common.ConnectionProperties) method.

      Josh Elser, What do you think?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                julianhyde Julian Hyde
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: