Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-1694

Clash in protobuf versions between Avatica and Hadoop

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.12.0
    • Component/s: avatica, pig-adapter
    • Labels:
      None

      Description

      When running Pig adapter tests, we get the following stack:

      java.lang.NoClassDefFoundError: com/google/protobuf/GeneratedMessageV3
      
      	at java.lang.ClassLoader.defineClass1(Native Method)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
      	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
      	at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
      	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      	at org.apache.calcite.avatica.ConnectionPropertiesImpl.<clinit>(ConnectionPropertiesImpl.java:37)
      	at org.apache.calcite.avatica.MetaImpl.<init>(MetaImpl.java:71)
      	at org.apache.calcite.jdbc.CalciteMetaImpl.<init>(CalciteMetaImpl.java:82)
      	at org.apache.calcite.jdbc.Driver.createMeta(Driver.java:169)
      	at org.apache.calcite.avatica.AvaticaConnection.<init>(AvaticaConnection.java:118)
      	at org.apache.calcite.jdbc.CalciteConnectionImpl.<init>(CalciteConnectionImpl.java:113)
      	at org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection.<init>(CalciteJdbc41Factory.java:114)
      	at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:59)
      	at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:44)
      	at org.apache.calcite.jdbc.CalciteFactory.newConnection(CalciteFactory.java:53)
      	at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
      	at java.sql.DriverManager.getConnection(DriverManager.java:664)
      	at java.sql.DriverManager.getConnection(DriverManager.java:208)
      	at org.apache.calcite.test.CalciteAssert$MapConnectionFactory.createConnection(CalciteAssert.java:1151)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.createConnection(CalciteAssert.java:1189)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1260)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1350)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1345)
      	at org.apache.calcite.test.PigAdapterTest.testImplWithGroupByMultipleFields(PigAdapterTest.java:116)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:483)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
      	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
      	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
      	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
      	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:483)
      	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
      Caused by: java.lang.ClassNotFoundException: com.google.protobuf.GeneratedMessageV3
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      	... 58 more
      

      followed by

      java.lang.NoClassDefFoundError: Could not initialize class org.apache.calcite.avatica.ConnectionPropertiesImpl
      
      	at org.apache.calcite.avatica.MetaImpl.<init>(MetaImpl.java:71)
      	at org.apache.calcite.jdbc.CalciteMetaImpl.<init>(CalciteMetaImpl.java:82)
      	at org.apache.calcite.jdbc.Driver.createMeta(Driver.java:169)
      	at org.apache.calcite.avatica.AvaticaConnection.<init>(AvaticaConnection.java:118)
      	at org.apache.calcite.jdbc.CalciteConnectionImpl.<init>(CalciteConnectionImpl.java:113)
      	at org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection.<init>(CalciteJdbc41Factory.java:114)
      	at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:59)
      	at org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:44)
      	at org.apache.calcite.jdbc.CalciteFactory.newConnection(CalciteFactory.java:53)
      	at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
      	at java.sql.DriverManager.getConnection(DriverManager.java:664)
      	at java.sql.DriverManager.getConnection(DriverManager.java:208)
      	at org.apache.calcite.test.CalciteAssert$MapConnectionFactory.createConnection(CalciteAssert.java:1151)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.createConnection(CalciteAssert.java:1189)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1260)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1350)
      	at org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1345)
      	at org.apache.calcite.test.PigAdapterTest.testImplWithMultipleFilters(PigAdapterTest.java:59)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:483)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
      	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
      	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
      	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
      	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:483)
      	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
      

      If you explicitly load ConnectionPropertiesImpl by adding the line

      Util.discard(org.apache.calcite.avatica.ConnectionPropertiesImpl.class);
      

      to PigSchemaFactory the error goes away.

      I think the problem might be that the Pig adapter pulls in Pig, which pulls in Hadoop, which depends on protobuf 2, whereas Avatica's ConnectionPropertiesImpl has some members that depend on protobuf 3.

      We can work around by having the Pig adapter load Calcite's MetaImpl (thereby avoiding an explicit dependency of the Pig adapter on Avatica).

      But the real solution, I think, is too reduce the dependencies of ConnectionPropertiesImpl. Calcite (and other Avatica users) do not care about the protobuf implementation of Avatica's wire protocol, but protobuf gets loaded due to the fromProto(Common.ConnectionProperties) method.

      Josh Elser, What do you think?

        Issue Links

          Activity

          Hide
          julianhyde Julian Hyde added a comment -

          Resolved in release 1.12.0 (2017-03-24).

          Show
          julianhyde Julian Hyde added a comment - Resolved in release 1.12.0 (2017-03-24).
          Hide
          julianhyde Julian Hyde added a comment -
          Show
          julianhyde Julian Hyde added a comment - Fixed Pig adapter in http://git-wip-us.apache.org/repos/asf/calcite/commit/256b1b90 . Thanks for the PR, Josh Elser !
          Hide
          julianhyde Julian Hyde added a comment -

          After Calcite 1.12 closes, I'll start a discussion on the dev list about Avatica release 1.10, when we should release it, and what people might be able to contribute. Personally I'd prefer to see an imperfect release in 1 month rather than a perfect release in 2 months.

          Show
          julianhyde Julian Hyde added a comment - After Calcite 1.12 closes, I'll start a discussion on the dev list about Avatica release 1.10, when we should release it, and what people might be able to contribute. Personally I'd prefer to see an imperfect release in 1 month rather than a perfect release in 2 months.
          Hide
          elserj Josh Elser added a comment -

          Do you agree that that would be an ideal solution

          Yup, you and I are in agreement. Getting transport serialization details out of Avatica's API is key.

          I think this is also predicated on CALCITE-839, or, maybe more accurately, this separation would encompass CALCITE-839.

          Would you be willing and able to make this change in Avatica in the medium term, say the next month

          Willing, yes. Able, maybe . I might be able to get some time to pick this up in April – I also need to get time to finish up the Array type work. Perhaps I can get the time to do both...

          Show
          elserj Josh Elser added a comment - Do you agree that that would be an ideal solution Yup, you and I are in agreement. Getting transport serialization details out of Avatica's API is key. I think this is also predicated on CALCITE-839 , or, maybe more accurately, this separation would encompass CALCITE-839 . Would you be willing and able to make this change in Avatica in the medium term, say the next month Willing, yes. Able, maybe . I might be able to get some time to pick this up in April – I also need to get time to finish up the Array type work. Perhaps I can get the time to do both...
          Hide
          julianhyde Julian Hyde added a comment -

          I think we can live with the inconsistency for now. I added an <ignoredUnusedDeclaredDependency> element to your patch, and am about to commit it.

          Suppose that we refactor Avatica so that the main API (I don't want to use the word "core") does not include protobuf. The jar for that main API will not need to have shaded and non-shaded versions, because it doesn't depend on protobuf or guava. Calcite will be able to use just that main API, and avoid issues of shading. (I see that Calcite core uses avatica-server, but only in tests.)

          Do you agree that that would be an ideal solution? Would you be willing and able to make this change in Avatica in the medium term, say the next month? I would like to sort out this problem before Calcite 1.13, which is probably in May.

          Show
          julianhyde Julian Hyde added a comment - I think we can live with the inconsistency for now. I added an <ignoredUnusedDeclaredDependency> element to your patch, and am about to commit it. Suppose that we refactor Avatica so that the main API (I don't want to use the word "core") does not include protobuf. The jar for that main API will not need to have shaded and non-shaded versions, because it doesn't depend on protobuf or guava. Calcite will be able to use just that main API, and avoid issues of shading. (I see that Calcite core uses avatica-server, but only in tests.) Do you agree that that would be an ideal solution? Would you be willing and able to make this change in Avatica in the medium term, say the next month? I would like to sort out this problem before Calcite 1.13, which is probably in May.
          Hide
          elserj Josh Elser added a comment -

          it seems a bit arbitrary to make the pig adapter use the shaded version

          Is there any reason why Calcite cannot use the shaded version throughout?

          Nope, you're right, it is a little arbitrary. For "maximum" flexibility in downstream users, it's convenient to provide both shaded and non-shaded variants of artifacts.

          For cases like these, a shaded artifact makes consumption of Calcite (and its dependencies) a no-op to work around. But, for other cases where downstream wants to use some of the same dependencies which Calcite bundles, it can negatively bloat the size of a final build artifact.

          I'd argue that both variants would be good to produce (similar to what I have done in Avatica already – go figure), but just switching calcite-core over to use the Avatica shaded artifacts would be the easy change. I can help "maven" this up if you agree

          Show
          elserj Josh Elser added a comment - it seems a bit arbitrary to make the pig adapter use the shaded version Is there any reason why Calcite cannot use the shaded version throughout? Nope, you're right, it is a little arbitrary. For "maximum" flexibility in downstream users, it's convenient to provide both shaded and non-shaded variants of artifacts. For cases like these, a shaded artifact makes consumption of Calcite (and its dependencies) a no-op to work around. But, for other cases where downstream wants to use some of the same dependencies which Calcite bundles, it can negatively bloat the size of a final build artifact. I'd argue that both variants would be good to produce (similar to what I have done in Avatica already – go figure), but just switching calcite-core over to use the Avatica shaded artifacts would be the easy change. I can help "maven" this up if you agree
          Hide
          julianhyde Julian Hyde added a comment -

          That seems to work. However, it seems a bit arbitrary to make the pig adapter use the shaded version ("avatica") while core, druid, elasticsearch, file, etc. use the non-shaded version ("avatica-core"). Is there any reason why Calcite cannot use the shaded version throughout?

          Show
          julianhyde Julian Hyde added a comment - That seems to work. However, it seems a bit arbitrary to make the pig adapter use the shaded version ("avatica") while core, druid, elasticsearch, file, etc. use the non-shaded version ("avatica-core"). Is there any reason why Calcite cannot use the shaded version throughout?
          Show
          elserj Josh Elser added a comment - Julian Hyde , can you try out HEAD of https://github.com/joshelser/calcite/tree/1694-pig-avatica-protobuf-dep?
          Hide
          elserj Josh Elser added a comment -

          But the real solution, I think, is too reduce the dependencies of ConnectionPropertiesImpl. Calcite (and other Avatica users) do not care about the protobuf implementation of Avatica's wire protocol, but protobuf gets loaded due to the fromProto(Common.ConnectionProperties) method.

          Agreed – it's been on my back-burner to separate out the "transports" to their own respective modules from "avatica-core".

          However.. Caused by: java.lang.ClassNotFoundException: com.google.protobuf.GeneratedMessageV3, this looks like the non-shaded Avatica artifacts are being used (again). We ran into this a few months back. Let me look at the calcite-pig dependency graph.

          Show
          elserj Josh Elser added a comment - But the real solution, I think, is too reduce the dependencies of ConnectionPropertiesImpl. Calcite (and other Avatica users) do not care about the protobuf implementation of Avatica's wire protocol, but protobuf gets loaded due to the fromProto(Common.ConnectionProperties) method. Agreed – it's been on my back-burner to separate out the "transports" to their own respective modules from "avatica-core". However.. Caused by: java.lang.ClassNotFoundException: com.google.protobuf.GeneratedMessageV3 , this looks like the non-shaded Avatica artifacts are being used (again). We ran into this a few months back. Let me look at the calcite-pig dependency graph.

            People

            • Assignee:
              Unassigned
              Reporter:
              julianhyde Julian Hyde
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development