Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 6.0
    • Component/s: None
    • Labels:
      None

      Issue Links

        Activity

        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        Turns out we are limited in the Guava version we can use by our kite morphlines dependency.

        Show
        markrmiller@gmail.com Mark Miller added a comment - Turns out we are limited in the Guava version we can use by our kite morphlines dependency.
        Hide
        dweiss Dawid Weiss added a comment -

        What do you mean by kite morphlines, Mark? I remember morfologik has a guava dependency, but this is only for constructing dictionaries, not for using them at runtime.

        Show
        dweiss Dawid Weiss added a comment - What do you mean by kite morphlines, Mark? I remember morfologik has a guava dependency, but this is only for constructing dictionaries, not for using them at runtime.
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        https://github.com/kite-sdk/kite/tree/master/kite-morphlines

        It's used for ETL with the new map-reduce contrib module.

        Show
        markrmiller@gmail.com Mark Miller added a comment - https://github.com/kite-sdk/kite/tree/master/kite-morphlines It's used for ETL with the new map-reduce contrib module.
        Hide
        whoschek wolfgang hoschek added a comment -

        What exactly is failing for you? morphlines was designed to run fine with any guava version >= 11.0.2. At least it did last I checked...

        Show
        whoschek wolfgang hoschek added a comment - What exactly is failing for you? morphlines was designed to run fine with any guava version >= 11.0.2. At least it did last I checked...
        Hide
        whoschek wolfgang hoschek added a comment -

        As mentioned above, morphlines was designed to run fine with any guava version >= 11.0.2.

        But the hadoop task tracker always puts guava 11.0.2 on the classpath of any MR job that it executes, so solr-mapreduce would need to figure out how to override or reorder that.

        Show
        whoschek wolfgang hoschek added a comment - As mentioned above, morphlines was designed to run fine with any guava version >= 11.0.2. But the hadoop task tracker always puts guava 11.0.2 on the classpath of any MR job that it executes, so solr-mapreduce would need to figure out how to override or reorder that.
        Hide
        ab Andrzej Bialecki added a comment -

        in MR2 you can override this ordering by setting mapreduce.job.user.classpath.first=true.

        Show
        ab Andrzej Bialecki added a comment - in MR2 you can override this ordering by setting mapreduce.job.user.classpath.first=true.
        Hide
        thetaphi Uwe Schindler added a comment -

        Move issue to Solr 4.9.

        Show
        thetaphi Uwe Schindler added a comment - Move issue to Solr 4.9.
        Hide
        elyograg Shawn Heisey added a comment -

        A user wants to use a library with Solr 4.8.0 that requires Guava 18. If I upgrade guava in the ivy versions file to 18.0, Solr will no longer compile.

        Show
        elyograg Shawn Heisey added a comment - A user wants to use a library with Solr 4.8.0 that requires Guava 18. If I upgrade guava in the ivy versions file to 18.0, Solr will no longer compile.
        Hide
        dweiss Dawid Weiss added a comment -

        Yeah, that Guava is severely outdated by now. I'll look into it again, see if I can bring it up to date a bit.

        For what it's worth, the user on solr-user wanted to run Lingo3G with Solr and this wouldn't help him anyway because there'd still be clashing HPPC versions... JAR hell.

        Show
        dweiss Dawid Weiss added a comment - Yeah, that Guava is severely outdated by now. I'll look into it again, see if I can bring it up to date a bit. For what it's worth, the user on solr-user wanted to run Lingo3G with Solr and this wouldn't help him anyway because there'd still be clashing HPPC versions... JAR hell.
        Hide
        dweiss Dawid Weiss added a comment -

        Ran the test suite and I get various exceptions from HDFS (9 in total).

        [15:42:07.466] ERROR   1.41s J0 | TestRecoveryHdfs.testRecoveryMultipleLogs <<<
           > Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.util.JvmPauseMonito
        r$Monitor
           >    at __randomizedtesting.SeedInfo.seed([531D0647CBD708B2]:0)
           >    at org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:175)
        
          2> 240555 ERROR (SUITE-TestRecoveryHdfs-seed#[531D0647CBD708B2]-worker) [    x:collection1] o.a.h.m.l.MethodMetric Error invoking method getBlocksTotal
           2> Caused by: java.lang.NullPointerException
          2>    at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
          2>    at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
          2>    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
        
        Show
        dweiss Dawid Weiss added a comment - Ran the test suite and I get various exceptions from HDFS (9 in total). [15:42:07.466] ERROR 1.41s J0 | TestRecoveryHdfs.testRecoveryMultipleLogs <<< > Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.util.JvmPauseMonito r$Monitor > at __randomizedtesting.SeedInfo.seed([531D0647CBD708B2]:0) > at org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:175) 2> 240555 ERROR (SUITE-TestRecoveryHdfs-seed#[531D0647CBD708B2]-worker) [ x:collection1] o.a.h.m.l.MethodMetric Error invoking method getBlocksTotal 2> Caused by: java.lang.NullPointerException 2> at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198) 2> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291) 2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
        Hide
        mdrob Mike Drob added a comment -

        Yea, this has been a problem for ages. Hadoop uses Guava 11.0.2, which will have compatibility issues with newer versions. HADOOP-11656 exists to try and alleviate the issue, but I don't see that landing soon. Not sure what to suggest for fixing this locally in the meanwhile.

        Show
        mdrob Mike Drob added a comment - Yea, this has been a problem for ages. Hadoop uses Guava 11.0.2, which will have compatibility issues with newer versions. HADOOP-11656 exists to try and alleviate the issue, but I don't see that landing soon. Not sure what to suggest for fixing this locally in the meanwhile.
        Hide
        dweiss Dawid Weiss added a comment -

        Yup, I found that issue too (linked to it from SOLR-7790). There are some options... using a shaded (repackaged) guava for the core Solr classes would be probably the most simple solution although in the long term I bet there would be two contribs that require two different versions... It'd be ideal to provide classloader separation between contribs and the core, but it looks like a major issue to me (in particular figuring out how to organize the build system around that...).

        Eh.

        Show
        dweiss Dawid Weiss added a comment - Yup, I found that issue too (linked to it from SOLR-7790 ). There are some options... using a shaded (repackaged) guava for the core Solr classes would be probably the most simple solution although in the long term I bet there would be two contribs that require two different versions... It'd be ideal to provide classloader separation between contribs and the core, but it looks like a major issue to me (in particular figuring out how to organize the build system around that...). Eh.
        Hide
        mdrob Mike Drob added a comment -

        The ideal solution is that we don't leak internal dependencies to clients, right? Such that Solr can use whichever version of Guava it needs, and clients can too, and same for all the other dependencies. So we would want our own version of the Hadoop classpath isolation. This isn't a unique problem to solve, nor is it an easy one.

        using a shaded (repackaged) guava for the core Solr classes would be probably the most simple solution although in the long term I bet there would be two contribs that require two different versions

        Why is that an issue? If we shade guava into solr-core, then contrib-1 can use "real" guava X and contrib-2 can use "real" guava Y. Neither one should be using the shaded internal guava.

        Show
        mdrob Mike Drob added a comment - The ideal solution is that we don't leak internal dependencies to clients, right? Such that Solr can use whichever version of Guava it needs, and clients can too, and same for all the other dependencies. So we would want our own version of the Hadoop classpath isolation. This isn't a unique problem to solve, nor is it an easy one. using a shaded (repackaged) guava for the core Solr classes would be probably the most simple solution although in the long term I bet there would be two contribs that require two different versions Why is that an issue? If we shade guava into solr-core, then contrib-1 can use "real" guava X and contrib-2 can use "real" guava Y. Neither one should be using the shaded internal guava.
        Hide
        dweiss Dawid Weiss added a comment -

        The ideal solution is that we don't leak internal dependencies to clients, right

        Yes, this would be ideal.

        This isn't a unique problem to solve, nor is it an easy one.

        Sadly, I know.

        If we shade guava into solr-core, then contrib-1 can use "real" guava X and contrib-2 can use "real" guava Y.

        My initial assessment didn't take into account that hdfs libraries are already part of Solr's lib (and thus common classloader). So there's no easy way to eradicate that old guava unless you repackage all the JARs in WEB-INF/lib.

        Also, there's a single class loader is per core (not per contrib) so any JARs from contribs are loaded into the same class loader. (That's what the code looks like in MemClassLoader and SolrCore... correct me if I'm wrong).

        Show
        dweiss Dawid Weiss added a comment - The ideal solution is that we don't leak internal dependencies to clients, right Yes, this would be ideal. This isn't a unique problem to solve, nor is it an easy one. Sadly, I know. If we shade guava into solr-core, then contrib-1 can use "real" guava X and contrib-2 can use "real" guava Y. My initial assessment didn't take into account that hdfs libraries are already part of Solr's lib (and thus common classloader). So there's no easy way to eradicate that old guava unless you repackage all the JARs in WEB-INF/lib. Also, there's a single class loader is per core (not per contrib) so any JARs from contribs are loaded into the same class loader. (That's what the code looks like in MemClassLoader and SolrCore... correct me if I'm wrong).
        Hide
        steve_rowe Steve Rowe added a comment -

        Now that the kite/morphlines contribs have been removed (SOLR-9221), can this be reconsidered? Or closed?

        Show
        steve_rowe Steve Rowe added a comment - Now that the kite/morphlines contribs have been removed ( SOLR-9221 ), can this be reconsidered? Or closed?

          People

          • Assignee:
            Unassigned
            Reporter:
            markrmiller@gmail.com Mark Miller
          • Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:

              Development