Solr
  1. Solr
  2. SOLR-3690

binary solr package isn't including lucene test-framework jar

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 4.0-ALPHA
    • Fix Version/s: 4.0, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      As noted by Koorosh on the solr-user list...

      I have been developing extensions to SOLR code using 4.0 truck. For JUnit testing I am extending AbstractSolrTestCase which in the ALPHA release is located in JAR apache-solr-test-framework-4.0.0-ALPHA.jar. However, this class extends LuceneTestCase which comes from JAR lucene-test-framework-4.0-SNAPSHOT.jar. In the ALPHA release the later JAR is not shipped or I can't find it.

      ...i confirmed that Solr 4.0.0-ALPHA binary package the solr test framework is in test./dist/apache-solr-test-framework-4.0.0-ALPHA.jar but the lucene test-framework jar isn't included at all.

      1. SOLR-3690.broken.jdoc.links.to.solrcore.patch
        11 kB
        Hoss Man
      2. SOLR-3690.patch
        10 kB
        Hoss Man
      3. SOLR-3690.patch
        10 kB
        Hoss Man

        Activity

        Hide
        Hoss Man added a comment -

        This sort of requires some careful thining baout how this should be packaged.

        Normally if a lucene jar is a solr core dependency, we include it in the war and that's it.

        If it's not a core dependency and it's only used by contribs, we ship it in the lucene-libs dir of that contrib.

        solr-test-framework isn't either of those though – and doesn't really have a place for a "lucene-lib" dir at the moment – but there's no reason we couldn't add test-framework/lucene-libs to the packages i suppose. probably a good idea to include a test-framework/README.txt anyway.

        Show
        Hoss Man added a comment - This sort of requires some careful thining baout how this should be packaged. Normally if a lucene jar is a solr core dependency, we include it in the war and that's it. If it's not a core dependency and it's only used by contribs, we ship it in the lucene-libs dir of that contrib. solr-test-framework isn't either of those though – and doesn't really have a place for a "lucene-lib" dir at the moment – but there's no reason we couldn't add test-framework/lucene-libs to the packages i suppose. probably a good idea to include a test-framework/README.txt anyway.
        Hide
        Robert Muir added a comment -

        It would be good to think about: ideally in the future we could imagine more lucene .jar's easily pluggable into Solr.

        So how should the thing be shipped?

        For instance, other lucene analysis jars are in the same boat as this lucene-test-framework.jar, they might
        be large (like lucene-test-framework which is 6MB!) and due to size excluded in the .war file, but still
        useful to some people.

        It would be nice to be consistent with the other analyzers and their dependencies (e.g. ICU or morfologik jars),
        the same way as lucene-test-framework and its dependencies (randomized-runner, etc). We might have more modules
        like this in the future (e.g. codecs is growing fast, we have a lot of similarities, these are easily pluggable today: just throwing these out there).

        Show
        Robert Muir added a comment - It would be good to think about: ideally in the future we could imagine more lucene .jar's easily pluggable into Solr. So how should the thing be shipped? For instance, other lucene analysis jars are in the same boat as this lucene-test-framework.jar, they might be large (like lucene-test-framework which is 6MB!) and due to size excluded in the .war file, but still useful to some people. It would be nice to be consistent with the other analyzers and their dependencies (e.g. ICU or morfologik jars), the same way as lucene-test-framework and its dependencies (randomized-runner, etc). We might have more modules like this in the future (e.g. codecs is growing fast, we have a lot of similarities, these are easily pluggable today: just throwing these out there).
        Hide
        Robert Muir added a comment -

        rmuir20120906-bulk-40-change

        Show
        Robert Muir added a comment - rmuir20120906-bulk-40-change
        Hide
        Hoss Man added a comment -

        It would be good to think about: ideally in the future we could imagine more lucene .jar's easily pluggable into Solr.

        right, this is what i was asking about in LUCENE-4044...

        FWIW: The other issue to consider here is how the packaging of these "uber-module-jars" will work in Solr binary pacakges ... right now it's the solr contribs that provide the factories that ensure the lucene modules they depend on make it into the final tgx/zip files ... if lucene modules start providing the factories themselves, we'll need some build.xml shenanigans to copy those module jars into the solr package dirs.

        ...but we punted on it there.

        It would be nice to be consistent with the other analyzers and their dependencies (e.g. ICU or morfologik jars), the same way as lucene-test-framework and its dependencies (randomized-runner, etc). We might have more modules

        Agreed. I think we should continue to punt on the larger issue (of including arbitrary lucene jars like codecs and what not) in the binary release, and focus here in this issue on treating the solr-test-framework the same as a solr-contrib and have it pull in it's lucene-libs and external dependencies via ivy.xml

        I'll start working on a patch

        Show
        Hoss Man added a comment - It would be good to think about: ideally in the future we could imagine more lucene .jar's easily pluggable into Solr. right, this is what i was asking about in LUCENE-4044 ... FWIW: The other issue to consider here is how the packaging of these "uber-module-jars" will work in Solr binary pacakges ... right now it's the solr contribs that provide the factories that ensure the lucene modules they depend on make it into the final tgx/zip files ... if lucene modules start providing the factories themselves, we'll need some build.xml shenanigans to copy those module jars into the solr package dirs. ...but we punted on it there. It would be nice to be consistent with the other analyzers and their dependencies (e.g. ICU or morfologik jars), the same way as lucene-test-framework and its dependencies (randomized-runner, etc). We might have more modules Agreed. I think we should continue to punt on the larger issue (of including arbitrary lucene jars like codecs and what not) in the binary release, and focus here in this issue on treating the solr-test-framework the same as a solr-contrib and have it pull in it's lucene-libs and external dependencies via ivy.xml I'll start working on a patch
        Hide
        Robert Muir added a comment -

        that sounds good. then we get consistency. separately we can figure out what might be better for the future.

        Show
        Robert Muir added a comment - that sounds good. then we get consistency. separately we can figure out what might be better for the future.
        Hide
        Hoss Man added a comment -

        Ok, patch ready for review.

        a few things to note:

        1) adding the logic to solr-test-framework to know about lucene-libs and libs caused some javadoc errors due to multiple copies of junit jars sowing up in the classpath used to build the javadocs ... but the real cruz of hte problem seemed to be that solr-test-framework was previously juping through unneccessary hoops to build the javadocs because it didn't use lucene-libs and libs properly ... i think i've fixed it appropriately.

        2) since solr-test-framework is not truly a contrib, i didn't put it's resulting README.txt, lucene-libs and lib dirs under "contrib" in the binary package, but i also didn't really think it made sense to pollute the top level dir with another directory either, so i followed the lead of solrj and put it under "dist/"

        comments?

        Show
        Hoss Man added a comment - Ok, patch ready for review. a few things to note: 1) adding the logic to solr-test-framework to know about lucene-libs and libs caused some javadoc errors due to multiple copies of junit jars sowing up in the classpath used to build the javadocs ... but the real cruz of hte problem seemed to be that solr-test-framework was previously juping through unneccessary hoops to build the javadocs because it didn't use lucene-libs and libs properly ... i think i've fixed it appropriately. 2) since solr-test-framework is not truly a contrib, i didn't put it's resulting README.txt, lucene-libs and lib dirs under "contrib" in the binary package, but i also didn't really think it made sense to pollute the top level dir with another directory either, so i followed the lead of solrj and put it under "dist/" comments?
        Hide
        Robert Muir added a comment -

        I can explain the screwed up unnecessary javadocs hoops. really this deserves a comment.

        With your current patch the 'hack' will break:

        javadocs-lint:
             [exec] 
             [exec] Crawl/parse...
             [exec] 
             [exec] Verify...
             [exec] 
             [exec] build/docs/api/test-framework/org/apache/solr/analysis/MockTokenizerFactory.html
             [exec]   BROKEN LINK: file:/home/rmuir/workspace/lucene-trunk/lucene/build/docs/test-framework/org/apache/lucene/analysis/Tokenizer.html
             [exec]   BROKEN LINK: file:/home/rmuir/workspace/lucene-trunk/lucene/build/docs/test-framework/org/apache/lucene/analysis/Tokenizer.html
             [exec] 
             [exec] build/docs/api/test-framework/org/apache/solr/analysis/MockCharFilterFactory.html
             [exec]   BROKEN LINK: file:/home/rmuir/workspace/lucene-trunk/lucene/build/docs/test-framework/org/apache/lucene/analysis/CharFilter.html
             [exec]   BROKEN LINK: file:/home/rmuir/workspace/lucene-trunk/lucene/build/docs/test-framework/org/apache/lucene/analysis/CharFilter.html
             [exec] 
             [exec] build/docs/api/test-framework/org/apache/solr/core/MockDirectoryFactory.html
             [exec]   BROKEN LINK: file:/home/rmuir/workspace/lucene-trunk/lucene/build/docs/test-framework/org/apache/lucene/store/Directory.html
             [exec]   BROKEN LINK: file:/home/rmuir/workspace/lucene-trunk/lucene/build/docs/test-framework/org/apache/lucene/store/Directory.html
             [exec] 
             [exec] build/docs/api/test-framework/org/apache/solr/core/MockFSDirectoryFactory.html
             [exec]   BROKEN LINK: file:/home/rmuir/workspace/lucene-trunk/lucene/build/docs/test-framework/org/apache/lucene/store/Directory.html
             [exec]   BROKEN LINK: file:/home/rmuir/workspace/lucene-trunk/lucene/build/docs/test-framework/org/apache/lucene/store/Directory.html
             [exec] 
             [exec] Broken javadocs links were found!
        
        BUILD FAILED
        /home/rmuir/workspace/lucene-trunk/solr/build.xml:528: The following error occurred while executing this line:
        /home/rmuir/workspace/lucene-trunk/lucene/common-build.xml:1740: exec returned: 1
        
        Show
        Robert Muir added a comment - I can explain the screwed up unnecessary javadocs hoops. really this deserves a comment. With your current patch the 'hack' will break: javadocs-lint: [exec] [exec] Crawl/parse... [exec] [exec] Verify... [exec] [exec] build/docs/api/test-framework/org/apache/solr/analysis/MockTokenizerFactory.html [exec] BROKEN LINK: file:/home/rmuir/workspace/lucene-trunk/lucene/build/docs/test-framework/org/apache/lucene/analysis/Tokenizer.html [exec] BROKEN LINK: file:/home/rmuir/workspace/lucene-trunk/lucene/build/docs/test-framework/org/apache/lucene/analysis/Tokenizer.html [exec] [exec] build/docs/api/test-framework/org/apache/solr/analysis/MockCharFilterFactory.html [exec] BROKEN LINK: file:/home/rmuir/workspace/lucene-trunk/lucene/build/docs/test-framework/org/apache/lucene/analysis/CharFilter.html [exec] BROKEN LINK: file:/home/rmuir/workspace/lucene-trunk/lucene/build/docs/test-framework/org/apache/lucene/analysis/CharFilter.html [exec] [exec] build/docs/api/test-framework/org/apache/solr/core/MockDirectoryFactory.html [exec] BROKEN LINK: file:/home/rmuir/workspace/lucene-trunk/lucene/build/docs/test-framework/org/apache/lucene/store/Directory.html [exec] BROKEN LINK: file:/home/rmuir/workspace/lucene-trunk/lucene/build/docs/test-framework/org/apache/lucene/store/Directory.html [exec] [exec] build/docs/api/test-framework/org/apache/solr/core/MockFSDirectoryFactory.html [exec] BROKEN LINK: file:/home/rmuir/workspace/lucene-trunk/lucene/build/docs/test-framework/org/apache/lucene/store/Directory.html [exec] BROKEN LINK: file:/home/rmuir/workspace/lucene-trunk/lucene/build/docs/test-framework/org/apache/lucene/store/Directory.html [exec] [exec] Broken javadocs links were found! BUILD FAILED /home/rmuir/workspace/lucene-trunk/solr/build.xml:528: The following error occurred while executing this line: /home/rmuir/workspace/lucene-trunk/lucene/common-build.xml:1740: exec returned: 1
        Hide
        Robert Muir added a comment -

        OK I was looking to 'show' the example on the website docs, but that isnt going to work. Basically the problem here is we have:

        • org.apache.lucene.analysis package in lucene-core
        • org.apache.lucene.analysis package in lucene-test-framework
        • ... with o.a.l.store and so on

        so this 'package' conflict causes problems for the package-list, and solr-test-framework links to lucene-test-framework
        because it 'has these packages' (but so does lucene-core!).

        yes its a hack, and yes you can say its bogus we have the duplicated packages in multiple modules, HOWEVER this is tests.
        I think its ok to have 'greyed out links' from the solr-test-framework javadocs to these lucene-core things, because
        its fair we should be able to test pkg-private stuff

        However, there might be a better, cleaner way than the hack i had before: such as somehow explicitly disabling these packages
        or some javadocs option like that.

        Show
        Robert Muir added a comment - OK I was looking to 'show' the example on the website docs, but that isnt going to work. Basically the problem here is we have: org.apache.lucene.analysis package in lucene-core org.apache.lucene.analysis package in lucene-test-framework ... with o.a.l.store and so on so this 'package' conflict causes problems for the package-list, and solr-test-framework links to lucene-test-framework because it 'has these packages' (but so does lucene-core!). yes its a hack, and yes you can say its bogus we have the duplicated packages in multiple modules, HOWEVER this is tests. I think its ok to have 'greyed out links' from the solr-test-framework javadocs to these lucene-core things, because its fair we should be able to test pkg-private stuff However, there might be a better, cleaner way than the hack i had before: such as somehow explicitly disabling these packages or some javadocs option like that.
        Hide
        Hoss Man added a comment -

        so this 'package' conflict causes problems for the package-list, and solr-test-framework links to lucene-test-framework

        yeah, that makes total sense ... it ust wasn't clear before why it was doing what it was doing. (although there was still a lot of wonky and unnecessary classpath stuff)

        I think its ok to have 'greyed out links' from the solr-test-framework javadocs to these lucene-core things

        Unfortunately it also means we have greyed out links to the core-test-framework stuff as well, and people are who are learning about solr-test-framework baseclasses are probably going to need to know just as much about stuff like LuceneTestCase, but i don't really see a good solution to that, so oh well.

        i switched back to using invoke-javadoc directly and got things working with javadocs-lint. i went on and attempted to at least get the solr-test-framework javadocs linking to the solr-core javadocs, but ran into a problem that the relative paths between them are differnet between "build" and "dist" and javadocs-lint expects them to work in build, so i scraped that (but it's attached if anyone wants to take a crack at it)

        Show
        Hoss Man added a comment - so this 'package' conflict causes problems for the package-list, and solr-test-framework links to lucene-test-framework yeah, that makes total sense ... it ust wasn't clear before why it was doing what it was doing. (although there was still a lot of wonky and unnecessary classpath stuff) I think its ok to have 'greyed out links' from the solr-test-framework javadocs to these lucene-core things Unfortunately it also means we have greyed out links to the core-test-framework stuff as well, and people are who are learning about solr-test-framework baseclasses are probably going to need to know just as much about stuff like LuceneTestCase, but i don't really see a good solution to that, so oh well. i switched back to using invoke-javadoc directly and got things working with javadocs-lint. i went on and attempted to at least get the solr-test-framework javadocs linking to the solr-core javadocs, but ran into a problem that the relative paths between them are differnet between "build" and "dist" and javadocs-lint expects them to work in build, so i scraped that (but it's attached if anyone wants to take a crack at it)
        Hide
        Robert Muir added a comment -

        I understand the problem. its actually part of a larger problem for solr?

        Currently the solr javadocs in dist/ are really organized in a mess. 'javadocs-all' creates this massive javadocs of everything (core, solrj, contribs) and puts
        it in the output folder THEN solrj is redundantly duplicated on top of this. so you have e.g.
        http://lucene.apache.org/solr/api-4_0_0-ALPHA/org/apache/solr/client/solrj/package-summary.html
        but also
        http://lucene.apache.org/solr/api-4_0_0-ALPHA/solrj/org/apache/solr/client/solrj/package-summary.html

        This is not useful and not very navigable.

        Instead I think, like lucene, we should nuke javadocs-all completely and organize javadocs into subfolders (core, solrj, test-framework, dataimport, ...),
        that have the links to each other.

        We can then add a nice auto-generated index page like http://lucene.apache.org/core/4_0_0-ALPHA/index.html that lets you navigate them, thats
        also checked of course by the link-checkers and such.

        Uwe just started talking about doing this on SOLR-2747

        I know this is all beyond the scope of the issue, I'm just explaining why its crazy and what I think we should do to make it simpler.

        Show
        Robert Muir added a comment - I understand the problem. its actually part of a larger problem for solr? Currently the solr javadocs in dist/ are really organized in a mess. 'javadocs-all' creates this massive javadocs of everything (core, solrj, contribs) and puts it in the output folder THEN solrj is redundantly duplicated on top of this. so you have e.g. http://lucene.apache.org/solr/api-4_0_0-ALPHA/org/apache/solr/client/solrj/package-summary.html but also http://lucene.apache.org/solr/api-4_0_0-ALPHA/solrj/org/apache/solr/client/solrj/package-summary.html This is not useful and not very navigable. Instead I think, like lucene, we should nuke javadocs-all completely and organize javadocs into subfolders (core, solrj, test-framework, dataimport, ...), that have the links to each other. We can then add a nice auto-generated index page like http://lucene.apache.org/core/4_0_0-ALPHA/index.html that lets you navigate them, thats also checked of course by the link-checkers and such. Uwe just started talking about doing this on SOLR-2747 I know this is all beyond the scope of the issue, I'm just explaining why its crazy and what I think we should do to make it simpler.
        Hide
        Uwe Schindler added a comment -

        +1, I can help

        Show
        Uwe Schindler added a comment - +1, I can help
        Hide
        Hoss Man added a comment -

        Committed revision 1373553. - trunk
        Committed revision 1373557. - 4x

        ...we can iterate on javadoc improvements elsewhere

        Show
        Hoss Man added a comment - Committed revision 1373553. - trunk Committed revision 1373557. - 4x ...we can iterate on javadoc improvements elsewhere
        Hide
        Uwe Schindler added a comment -

        Closed after release.

        Show
        Uwe Schindler added a comment - Closed after release.

          People

          • Assignee:
            Hoss Man
            Reporter:
            Hoss Man
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development