Solr
  1. Solr
  2. SOLR-2659

src/test-files/** should be moved under src/test-files/<module-name>/** for all Solr modules except core

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 3.4, 4.0-ALPHA
    • Fix Version/s: 3.4, 4.0-ALPHA
    • Component/s: Build
    • Labels:
      None

      Description

      SOLR-2452 split the solrj & common tests and test-files out from under solr/src/test{,-files} and placed them under solr/solrj/.

      Because IntelliJ's dependency scheme can't directly support the dependencies among the core/, solrj/, and test-framework/ internal modules, IntelliJ runs core/ and solrj/ tests under the monolithic IntelliJ "solr" module,

      As a result, when IntelliJ copies core/src/test-files/** and solrj/src/test-files/** to solr/build/solr-idea/classes/test/ (the test output directory), only one file from each same-named file pair can reside in the target directory, e.g. solr/conf/schema.xml. When same-named files differ between the two test-files/ directories, tests will fail. E.g.: LUCENE-2048 introduced a nopositions fieldType and a nopositionstext field into core/src/test-files/solr/conf/schema.xml, but not into the same-named file under solrj/src/test-files/, so when IntelliJ chooses the solrj version when copying resources, the core test that depends on the nopositionstext field (TestOmitPositions) will fail.

      I propose adding an extra directory level under solrj/src/test-files/: solrj/src/test-files/solrj/. That way, files from core/src/test-files/ can have the same names, but still co-exist when copied to the test output directory by IntelliJ.

      To maintain consistency, as well as avoid future naming conflicts, all other solr modules except core should switch to the same layout: src/test-files/<module-name>/*. Currently all contribs' solr homes are named src/test-files/solr-<module-name>/ - these directories should be renamed to src/test-files/<module-name>/solr.

      1. SOLR-2659.part2.patch
        2 kB
        Steve Rowe
      2. SOLR-2659.patch
        30 kB
        Steve Rowe
      3. SOLR-2659.patch
        9 kB
        Steve Rowe
      4. SOLR-2659.svn.movement.sh
        1 kB
        Steve Rowe

        Activity

        Steve Rowe created issue -
        Hide
        Steve Rowe added a comment -

        Patch implementing the proposal.

        Before applying the patch, run this svn movement script on the cmdline (under a Bash shell):

        svn mkdir solr/solrj/src/test-files/solrj
        svn mv solr/solrj/src/test-files/{README,books.csv,docs1.xml,docs2.xml,sampleDateFacetResponse.xml,solr} solr/solrj/src/test-files/solrj
        
        Show
        Steve Rowe added a comment - Patch implementing the proposal. Before applying the patch, run this svn movement script on the cmdline (under a Bash shell): svn mkdir solr/solrj/src/test-files/solrj svn mv solr/solrj/src/test-files/{README,books.csv,docs1.xml,docs2.xml,sampleDateFacetResponse.xml,solr} solr/solrj/src/test-files/solrj
        Steve Rowe made changes -
        Field Original Value New Value
        Attachment SOLR-2659.patch [ 12486770 ]
        Hide
        Robert Muir added a comment -

        maybe simpler/more consistent if its solr home is just solr-solrj? this is how the other ones work, e.g. solr-dih, solr-extraction, ...

        Show
        Robert Muir added a comment - maybe simpler/more consistent if its solr home is just solr-solrj? this is how the other ones work, e.g. solr-dih, solr-extraction, ...
        Hide
        Steve Rowe added a comment -

        maybe simpler/more consistent if its solr home is just solr-solrj? this is how the other ones work, e.g. solr-dih, solr-extraction, ...

        Unlike all of these, however, solrj/src/test-files/ has files that are at the top level, not under its solr home.

        Show
        Steve Rowe added a comment - maybe simpler/more consistent if its solr home is just solr-solrj? this is how the other ones work, e.g. solr-dih, solr-extraction, ... Unlike all of these, however, solrj/src/test-files/ has files that are at the top level, not under its solr home.
        Hide
        Steve Rowe added a comment -

        maybe simpler/more consistent if its solr home is just solr-solrj? this is how the other ones work, e.g. solr-dih, solr-extraction, ...

        Unlike all of these, however, solrj/src/test-files/ has files that are at the top level, not under its solr home.

        I was wrong: Solr contribs clustering, dih-extras, extraction, and uima all have files at the top level not under */src/test-files/solr-*/. These appear mostly to have different names, but in at least one case (solr-word.pdf), there is a same-named file in more than one module (extraction and dih-extras). So it would appear that the systemic problem is not fixed by just renaming the solr homes?

        Show
        Steve Rowe added a comment - maybe simpler/more consistent if its solr home is just solr-solrj? this is how the other ones work, e.g. solr-dih, solr-extraction, ... Unlike all of these, however, solrj/src/test-files/ has files that are at the top level, not under its solr home. I was wrong: Solr contribs clustering, dih-extras, extraction, and uima all have files at the top level not under */src/test-files/solr-*/ . These appear mostly to have different names, but in at least one case ( solr-word.pdf ), there is a same-named file in more than one module (extraction and dih-extras). So it would appear that the systemic problem is not fixed by just renaming the solr homes?
        Hide
        Robert Muir added a comment -

        FYI i copied this file here because it used to 'reach' back into extraction to get it...

        Show
        Robert Muir added a comment - FYI i copied this file here because it used to 'reach' back into extraction to get it...
        Hide
        Steve Rowe added a comment -

        Naming solr homes for the contrib in whose test-files dir they reside is an effort at namespacing: allowing each contrib to have its own version of same-named files. It seems to me that leaving files outside of the namespacing directory removes the freedom & protection it affords.

        This is exactly the reason I put everything under solrj/src/test-files/ into a solrj/ sub-directory: several of the top-level files have the same names as those under core/src/test-files/.

        Show
        Steve Rowe added a comment - Naming solr homes for the contrib in whose test-files dir they reside is an effort at namespacing: allowing each contrib to have its own version of same-named files. It seems to me that leaving files outside of the namespacing directory removes the freedom & protection it affords. This is exactly the reason I put everything under solrj/src/test-files/ into a solrj/ sub-directory: several of the top-level files have the same names as those under core/src/test-files/ .
        Hide
        Robert Muir added a comment -

        maybe we should do this for all the modules then?

        Show
        Robert Muir added a comment - maybe we should do this for all the modules then?
        Hide
        Steve Rowe added a comment -

        My vote is to do this for all the modules except Solr core, which can use the anonymous namespace (no namespacing directory under test-files/).

        Show
        Steve Rowe added a comment - My vote is to do this for all the modules except Solr core, which can use the anonymous namespace (no namespacing directory under test-files/ ).
        Hide
        Robert Muir added a comment -

        +1

        Show
        Robert Muir added a comment - +1
        Steve Rowe made changes -
        Summary IntelliJ resource copying of solrj/src/test-files/** and core/src/test-files/** to build output directory has to choose between/overwrite same-named files src/test-files/** should be moved under src/test-files/<module-name>/** for all Solr modules except core
        Description SOLR-2452 split the solrj & common tests and test-files out from under {{solr/src/test\{,-files\}}} and placed them under {{solr/solrj/}}.

        Because IntelliJ's dependency scheme can't directly support the dependencies among the {{core/}}, {{solrj/}}, and {{test-framework/}} internal modules, IntelliJ runs {{core/}} and {{solrj/}} tests under the monolithic IntelliJ "solr" module,

        As a result, when IntelliJ copies {{core/src/test-files/\*\*}} and {{solrj/src/test-files/\*\*}} to {{solr/build/solr-idea/classes/test/}} (the test output directory), only one file from each same-named file pair can reside in the target directory, e.g. {{solr/conf/schema.xml}}. When same-named files differ between the two {{test-files/}} directories, tests will fail. E.g.: LUCENE-2048 introduced a {{nopositions}} fieldType and a {{nopositionstext}} field into {{core/src/test-files/solr/conf/schema.xml}}, but not into the same-named file under {{solrj/src/test-files/}}, so when IntelliJ chooses the solrj version when copying resources, the core test that depends on the {{nopositionstext}} field ({{TestOmitPositions}}) will fail.

        I propose adding an extra directory level under {{solrj/src/test-files/}}: {{solrj/src/test-files/solrj/}}. That way, files from {{core/src/test-files/}} can have the same names, but still co-exist when copied to the test output directory by IntelliJ.
        SOLR-2452 split the solrj & common tests and test-files out from under {{solr/src/test\{,-files\}}} and placed them under {{solr/solrj/}}.

        Because IntelliJ's dependency scheme can't directly support the dependencies among the {{core/}}, {{solrj/}}, and {{test-framework/}} internal modules, IntelliJ runs {{core/}} and {{solrj/}} tests under the monolithic IntelliJ "solr" module,

        As a result, when IntelliJ copies {{core/src/test-files/\*\*}} and {{solrj/src/test-files/\*\*}} to {{solr/build/solr-idea/classes/test/}} (the test output directory), only one file from each same-named file pair can reside in the target directory, e.g. {{solr/conf/schema.xml}}. When same-named files differ between the two {{test-files/}} directories, tests will fail. E.g.: LUCENE-2048 introduced a {{nopositions}} fieldType and a {{nopositionstext}} field into {{core/src/test-files/solr/conf/schema.xml}}, but not into the same-named file under {{solrj/src/test-files/}}, so when IntelliJ chooses the solrj version when copying resources, the core test that depends on the {{nopositionstext}} field ({{TestOmitPositions}}) will fail.

        I propose adding an extra directory level under {{solrj/src/test-files/}}: {{solrj/src/test-files/solrj/}}. That way, files from {{core/src/test-files/}} can have the same names, but still co-exist when copied to the test output directory by IntelliJ.

        To maintain consistency, as well as avoid future naming conflicts, all other solr modules except core should switch to the same layout: {{src/test-files/<module-name>/\*}}. Currently all contribs' solr homes are named {{src/test-files/solr-<module-name>/}} - these directories should be renamed to {{src/test-files/<module-name>/solr}}.
        Hide
        Steve Rowe added a comment -

        Patch implementing changes, including modifications to the SOLR-2452 patch hacking script under dev-tools/scripts/.

        Run the attached svn movement script before applying the patch.

        Show
        Steve Rowe added a comment - Patch implementing changes, including modifications to the SOLR-2452 patch hacking script under dev-tools/scripts/. Run the attached svn movement script before applying the patch.
        Steve Rowe made changes -
        Attachment SOLR-2659.patch [ 12487405 ]
        Attachment SOLR-2659.svn.movement.sh [ 12487406 ]
        sarowe committed 1149600 (193 files)
        Reviews: none

        SOLR-2659: Moved src/test-files/** under src/test-files/<module-name>/** for all Solr modules except core

        Lucene trunk
        sarowe committed 1149601 (186 files)
        Reviews: none

        SOLR-2659: Moved src/test-files/** under src/test-files/<module-name>/** for all Solr modules except core

        Lucene branch_3x
        Hide
        Steve Rowe added a comment -

        Committed:

        • r1149600: trunk
        • r1149601: branch_3x
        Show
        Steve Rowe added a comment - Committed: r1149600: trunk r1149601: branch_3x
        Steve Rowe made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 3.4 [ 12316683 ]
        Fix Version/s 4.0 [ 12314992 ]
        Resolution Fixed [ 1 ]
        Hide
        Steve Rowe added a comment - - edited

        When running Solrj's tests, ExternalPaths.determineSourceHome() fails, e.g.:

        Build: https://builds.apache.org/job/Lucene-Solr-tests-only-3.x/9695/
        
        15 tests failed.
        FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.SolrExampleBinaryTest
        
        Error Message:
        null
        
        Stack Trace:
        java.lang.ExceptionInInitializerError
        	at org.apache.solr.client.solrj.SolrExampleBinaryTest.beforeTest(SolrExampleBinaryTest.java:36)
        Caused by: java.lang.RuntimeException: Cannot find resource: solr/conf
        	at org.apache.solr.SolrTestCaseJ4.getFile(SolrTestCaseJ4.java:1101)
        	at org.apache.solr.util.ExternalPaths.determineSourceHome(ExternalPaths.java:40)
        	at org.apache.solr.util.ExternalPaths.<clinit>(ExternalPaths.java:30)
        
        Show
        Steve Rowe added a comment - - edited When running Solrj's tests, ExternalPaths.determineSourceHome() fails, e.g.: Build: https://builds.apache.org/job/Lucene-Solr-tests-only-3.x/9695/ 15 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.SolrExampleBinaryTest Error Message: null Stack Trace: java.lang.ExceptionInInitializerError at org.apache.solr.client.solrj.SolrExampleBinaryTest.beforeTest(SolrExampleBinaryTest.java:36) Caused by: java.lang.RuntimeException: Cannot find resource: solr/conf at org.apache.solr.SolrTestCaseJ4.getFile(SolrTestCaseJ4.java:1101) at org.apache.solr.util.ExternalPaths.determineSourceHome(ExternalPaths.java:40) at org.apache.solr.util.ExternalPaths.<clinit>(ExternalPaths.java:30)
        Steve Rowe made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Steve Rowe added a comment -

        Patch that allows tests to pass.

        Committing shortly

        Show
        Steve Rowe added a comment - Patch that allows tests to pass. Committing shortly
        Steve Rowe made changes -
        Attachment SOLR-2659.part2.patch [ 12487461 ]
        sarowe committed 1149682 (2 files)
        Reviews: none

        SOLR-2659: Fix ExternalPaths.determineSourceHome() and SolrJettyTestBase.createJetty() to allow Solrj tests to pass.

        sarowe committed 1149691 (2 files)
        Hide
        Steve Rowe added a comment -

        Committed the part2 patch:

        • r1149682: trunk
        • r1149691: branch_3x

        I'll leave the issue open until Jenkins builds including Solr on both trunk and branch_3x have succeeded. (I triggered Solr-3.x and Solr-trunk builds on Jenkins, so successful builds should be finished within a couple of hours.)

        Show
        Steve Rowe added a comment - Committed the part2 patch: r1149682: trunk r1149691: branch_3x I'll leave the issue open until Jenkins builds including Solr on both trunk and branch_3x have succeeded. (I triggered Solr-3.x and Solr-trunk builds on Jenkins, so successful builds should be finished within a couple of hours.)
        Hide
        Steve Rowe added a comment -

        The Solr-3.x and Solr-trunk Jenkins builds have both succeeded.

        Show
        Steve Rowe added a comment - The Solr-3.x and Solr-trunk Jenkins builds have both succeeded.
        Steve Rowe made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Robert Muir added a comment -

        bulk close for 3.4

        Show
        Robert Muir added a comment - bulk close for 3.4
        Robert Muir made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Steve Rowe
            Reporter:
            Steve Rowe
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development