Lucene - Core
  1. Lucene - Core
  2. LUCENE-2657

Replace Maven POM templates with full POMs, and change documentation accordingly

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.1, 4.0-ALPHA
    • Fix Version/s: 3.1, 4.0-ALPHA
    • Component/s: general/build
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      The current Maven POM templates only contain dependency information, the bare bones necessary for uploading artifacts to the Maven repository.

      The full Maven POMs in the attached patch include the information necessary to run a multi-module Maven build, in addition to serving the same purpose as the current POM templates.

      Several dependencies are not available through public maven repositories. A profile in the top-level POM can be activated to install these dependencies from the various lib/ directories into your local repository. From the top-level directory:

      mvn -N -Pbootstrap install
      

      Once these non-Maven dependencies have been installed, to run all Lucene/Solr tests via Maven's surefire plugin, and populate your local repository with all artifacts, from the top level directory, run:

      mvn install
      

      When one Lucene/Solr module depends on another, the dependency is declared on the artifact(s) produced by the other module and deposited in your local repository, rather than on the other module's un-jarred compiler output in the build/ directory, so you must run mvn install on the other module before its changes are visible to the module that depends on it.

      To create all the artifacts without running tests:

      mvn -DskipTests install
      

      I almost always include the clean phase when I do a build, e.g.:

      mvn -DskipTests clean install
      
      1. LUCENE-2657.patch
        214 kB
        Steve Rowe
      2. LUCENE-2657.patch
        203 kB
        Steve Rowe
      3. LUCENE-2657.patch
        202 kB
        Steve Rowe
      4. LUCENE-2657.patch
        232 kB
        Steve Rowe
      5. LUCENE-2657.patch
        217 kB
        Steve Rowe
      6. LUCENE-2657.patch
        220 kB
        Steve Rowe
      7. LUCENE-2657.patch
        220 kB
        Steve Rowe
      8. LUCENE-2657.patch
        194 kB
        Steve Rowe
      9. LUCENE-2657.patch
        198 kB
        Steve Rowe
      10. LUCENE-2657.patch
        198 kB
        Steve Rowe
      11. LUCENE-2657.patch
        198 kB
        Steve Rowe
      12. LUCENE-2657.patch
        233 kB
        Steve Rowe
      13. LUCENE-2657.patch
        228 kB
        Steve Rowe
      14. LUCENE-2657.patch
        227 kB
        Steve Rowe
      15. LUCENE-2657.patch
        229 kB
        Steve Rowe
      16. LUCENE-2657.patch
        222 kB
        Steve Rowe
      17. LUCENE-2657-branch_3x.patch
        217 kB
        Steve Rowe
      18. LUCENE-2657-branch_3x.patch
        206 kB
        Steve Rowe

        Issue Links

          Activity

          Hide
          Robert Muir added a comment - - edited

          How would the BuildListener interface know about dependencies? Does it have some magic way to know this?

          As an example, lets take modules/analysis/icu which has 3 dependencies:

          • Lucene core itself (implicit from contrib-build.xml)
          • external dependency: ICU
          • internal dependency: modules/analysis/common

          take a look at modules/analysis/icu's pom.xml which has:

            <dependencies>
              <dependency>
                <groupId>com.ibm.icu</groupId>
                <artifactId>icu4j</artifactId>
                <version>${icu-version}</version>
              </dependency>
            </dependencies>
          

          However, our ant builds (that depend on common-build/contrib-build) declare their dependencies in a semi-standard way:

          • External dependencies:
            <path id="additional.dependencies">
              <fileset dir="lib" includes="icu4j-*.jar"/>
            </path>
            
          • Internal dependencies:
            <module-uptodate name="analysis/common" 
            jarfile="../build/common/lucene-analyzers-common-${version}.jar"
              property="analyzers-common.uptodate" classpath.property="analyzers-common.jar"/>
            

          The contrib-build.xml already has a 'dist-maven' target, that is called recursively. Perhaps an alternative would be to improve contrib-build.xml for it to have a 'generate-maven' target, also called recursively.
          I've already prototyped/proposed in SOLR-2002 that we migrate the solr build to extend the lucene build, so everywhere would use it.

          Furthermore, couldnt we also make a recursive 'test-maven' target, that generates a maven project to 'download' or whatever it needs, then tries to run all the tests?
          If somehow the maven is broken, the tests simply won't pass.

          I realize that running all of a modules tests again redundantly via 'maven' might not be the most elegant solution, but it seems like it would test that everything is working.

          Show
          Robert Muir added a comment - - edited How would the BuildListener interface know about dependencies? Does it have some magic way to know this? As an example, lets take modules/analysis/icu which has 3 dependencies: Lucene core itself (implicit from contrib-build.xml) external dependency: ICU internal dependency: modules/analysis/common take a look at modules/analysis/icu's pom.xml which has: <dependencies> <dependency> <groupId>com.ibm.icu</groupId> <artifactId>icu4j</artifactId> <version>${icu-version}</version> </dependency> </dependencies> However, our ant builds (that depend on common-build/contrib-build) declare their dependencies in a semi-standard way: External dependencies: <path id="additional.dependencies"> <fileset dir="lib" includes="icu4j-*.jar"/> </path> Internal dependencies: <module-uptodate name="analysis/common" jarfile="../build/common/lucene-analyzers-common-${version}.jar" property="analyzers-common.uptodate" classpath.property="analyzers-common.jar"/> The contrib-build.xml already has a 'dist-maven' target, that is called recursively. Perhaps an alternative would be to improve contrib-build.xml for it to have a 'generate-maven' target, also called recursively. I've already prototyped/proposed in SOLR-2002 that we migrate the solr build to extend the lucene build, so everywhere would use it. Furthermore, couldnt we also make a recursive 'test-maven' target, that generates a maven project to 'download' or whatever it needs, then tries to run all the tests? If somehow the maven is broken, the tests simply won't pass. I realize that running all of a modules tests again redundantly via 'maven' might not be the most elegant solution, but it seems like it would test that everything is working.
          Hide
          Steve Rowe added a comment -

          How would the BuildListener interface know about dependencies? Does it have some magic way to know this?

          BuildListener has hooks for build task onset and completion events (inter alia). ant2ide listens for Javac task completion, and captures from it the source and target directories, as well as the build classpath. You have to invoke compilation from an Ant build in order for this to work.

          Seems kinda magical to me

          The missing part here is figuring out the maven groupId/artifactId/version, and I think this can be dealt with by looking at the manifest in the jar. Maven-produced jars also contain their POMs, and pulling from there would be even simpler.

          Show
          Steve Rowe added a comment - How would the BuildListener interface know about dependencies? Does it have some magic way to know this? BuildListener has hooks for build task onset and completion events (inter alia). ant2ide listens for Javac task completion, and captures from it the source and target directories, as well as the build classpath. You have to invoke compilation from an Ant build in order for this to work. Seems kinda magical to me The missing part here is figuring out the maven groupId/artifactId/version, and I think this can be dealt with by looking at the manifest in the jar. Maven-produced jars also contain their POMs, and pulling from there would be even simpler.
          Hide
          Robert Muir added a comment -

          BuildListener has hooks for build task onset and completion events (inter alia). ant2ide listens for Javac task completion, and captures from it the source and target directories, as well as the build classpath. You have to invoke compilation from an Ant build in order for this to work.

          Seems kinda magical to me

          OK, i guess this is similar to when I open the project up in eclipse and say 'from existing ant build.xml file' and it sets up my eclipse project.

          Show
          Robert Muir added a comment - BuildListener has hooks for build task onset and completion events (inter alia). ant2ide listens for Javac task completion, and captures from it the source and target directories, as well as the build classpath. You have to invoke compilation from an Ant build in order for this to work. Seems kinda magical to me OK, i guess this is similar to when I open the project up in eclipse and say 'from existing ant build.xml file' and it sets up my eclipse project.
          Hide
          Chris Male added a comment -

          I'm wondering whether generating poms is just going to make them more mysterious. While they are templates I can at least see them and they can be validated by XML validators (including IDEs). Suddenly they are created through a more complex process which we also have to maintain.

          Assuming we have well-structured poms in the first place (huge assumption, different issue), are developers unprepared to add the appropriate dependency to the necessary pom? It doesn't seem like it'll happen very often, most of our artifacts have very few dependencies and we resist increasing them. We can document the process, make it very clear exactly what needs to be added and where.

          Also, I'm not sure all jars are going to have the appropriate manifest. If the pom is generated (like Artifactory can do btw), I'm not sure it is packaged in the jar?

          I think Robert was onto the right idea with this original proposal: If we consider our modules section and if we add a pom at the modules level and then one for each module, running a build from the modules level will recursively build each module. If there is something wrong with the poms themselves, maven will fail. If a dependency is omitted meaning the code cannot be built, maven will fail. If any of these 2 issues occur, then the build would fail and an issue can be created for those more comfortable with maven to fix.

          Show
          Chris Male added a comment - I'm wondering whether generating poms is just going to make them more mysterious. While they are templates I can at least see them and they can be validated by XML validators (including IDEs). Suddenly they are created through a more complex process which we also have to maintain. Assuming we have well-structured poms in the first place (huge assumption, different issue), are developers unprepared to add the appropriate dependency to the necessary pom? It doesn't seem like it'll happen very often, most of our artifacts have very few dependencies and we resist increasing them. We can document the process, make it very clear exactly what needs to be added and where. Also, I'm not sure all jars are going to have the appropriate manifest. If the pom is generated (like Artifactory can do btw), I'm not sure it is packaged in the jar? I think Robert was onto the right idea with this original proposal: If we consider our modules section and if we add a pom at the modules level and then one for each module, running a build from the modules level will recursively build each module. If there is something wrong with the poms themselves, maven will fail. If a dependency is omitted meaning the code cannot be built, maven will fail. If any of these 2 issues occur, then the build would fail and an issue can be created for those more comfortable with maven to fix.
          Hide
          Steve Rowe added a comment -

          Assuming we have well-structured poms in the first place (huge assumption, different issue), are developers unprepared to add the appropriate dependency to the necessary pom?

          If nothing else, it is abundantly clear that there are more than a couple of Lucene/Solr developers that don't want to put in any effort for Maven's sake.

          Also, I'm not sure all jars are going to have the appropriate manifest. If the pom is generated (like Artifactory can do btw), I'm not sure it is packaged in the jar?

          I've looked into 10-15 jars, and very few have the required info. Some even have the wrong information in the manifest...

          I think we can use .jar digest (SHA-1) as identifiers to look up the appropriate Maven info to automatically set up POM dependencies. I wrote a script to crawl the central Maven repo for SHA-1 keyed Maven coordinates (groupId:artifactId:version), and it seems to work, but at the rate it's progressing, it will take 4 days to finish. Anyway, it seems like overkill to gather info on all of the ~250k artifacts in the Maven central repo, just to handle the ~80 .jar files in the Lucene/Solr source tree.

          I found a free service that provides Maven central repo search, including by artifact digest: http://www.jarvana.com/jarvana/search - it uses Lucene to power the search . I plan on using this as the basis for automatic Maven coordinate lookup for arbitrary .jars.

          Show
          Steve Rowe added a comment - Assuming we have well-structured poms in the first place (huge assumption, different issue), are developers unprepared to add the appropriate dependency to the necessary pom? If nothing else, it is abundantly clear that there are more than a couple of Lucene/Solr developers that don't want to put in any effort for Maven's sake. Also, I'm not sure all jars are going to have the appropriate manifest. If the pom is generated (like Artifactory can do btw), I'm not sure it is packaged in the jar? I've looked into 10-15 jars, and very few have the required info. Some even have the wrong information in the manifest... I think we can use .jar digest (SHA-1) as identifiers to look up the appropriate Maven info to automatically set up POM dependencies. I wrote a script to crawl the central Maven repo for SHA-1 keyed Maven coordinates (groupId:artifactId:version), and it seems to work, but at the rate it's progressing, it will take 4 days to finish. Anyway, it seems like overkill to gather info on all of the ~250k artifacts in the Maven central repo, just to handle the ~80 .jar files in the Lucene/Solr source tree. I found a free service that provides Maven central repo search, including by artifact digest: http://www.jarvana.com/jarvana/search - it uses Lucene to power the search . I plan on using this as the basis for automatic Maven coordinate lookup for arbitrary .jars.
          Hide
          Chris Male added a comment -

          If nothing else, it is abundantly clear that there are more than a couple of Lucene/Solr developers that don't want to put in any effort for Maven's sake.

          If developers are willing to add the dependency jar and lines to the ant file but not add the 5 lines to the pom, then I'm not sure any of these attempts are really going to succeed. Sure many developers don't like maven, but I haven't seen any straight out refuse to use it yet (unless I misread the large email thread), they are just demanding ease of use.

          My concern here is this process is going in the opposite direction by making it vastly more complicated. What your proposing sounds like a great app independent of Lucene - a way to convert from an ant build system to a maven one, but I'm unsure its something that should be part of a build process instead of a once off thing. Certainly when we are trying to simplify said build process.

          Show
          Chris Male added a comment - If nothing else, it is abundantly clear that there are more than a couple of Lucene/Solr developers that don't want to put in any effort for Maven's sake. If developers are willing to add the dependency jar and lines to the ant file but not add the 5 lines to the pom, then I'm not sure any of these attempts are really going to succeed. Sure many developers don't like maven, but I haven't seen any straight out refuse to use it yet (unless I misread the large email thread), they are just demanding ease of use. My concern here is this process is going in the opposite direction by making it vastly more complicated. What your proposing sounds like a great app independent of Lucene - a way to convert from an ant build system to a maven one, but I'm unsure its something that should be part of a build process instead of a once off thing. Certainly when we are trying to simplify said build process.
          Hide
          Steve Rowe added a comment -

          My concern here is this process is going in the opposite direction by making it vastly more complicated. What your proposing sounds like a great app independent of Lucene - a way to convert from an ant build system to a maven one, but I'm unsure its something that should be part of a build process instead of a once off thing. Certainly when we are trying to simplify said build process.

          It is not the build process that requires simplification (is anybody really calling for that?), but rather developers' interaction with it?

          Show
          Steve Rowe added a comment - My concern here is this process is going in the opposite direction by making it vastly more complicated. What your proposing sounds like a great app independent of Lucene - a way to convert from an ant build system to a maven one, but I'm unsure its something that should be part of a build process instead of a once off thing. Certainly when we are trying to simplify said build process. It is not the build process that requires simplification (is anybody really calling for that?), but rather developers' interaction with it?
          Hide
          Steve Rowe added a comment -

          I don't see this issue as providing a complete Ant->Maven conversion system. The poms I envision will look just like the POM templates after they are interpolated - they will provide Maven coordinates for artifacts and their dependencies, nothing more. And the process to produce them can be carried out as part of generating Maven artifacts. I don't think they will need to be checked in, but they certainly could be.

          In other words, I don't plan on building a complete Maven build configuration here. Just the bare-bones POMs required to upload artifacts, like the manually synched system we have now. Only without the manual synching part.

          Vive la simplification!

          Show
          Steve Rowe added a comment - I don't see this issue as providing a complete Ant->Maven conversion system. The poms I envision will look just like the POM templates after they are interpolated - they will provide Maven coordinates for artifacts and their dependencies, nothing more. And the process to produce them can be carried out as part of generating Maven artifacts. I don't think they will need to be checked in, but they certainly could be. In other words, I don't plan on building a complete Maven build configuration here. Just the bare-bones POMs required to upload artifacts, like the manually synched system we have now. Only without the manual synching part. Vive la simplification!
          Hide
          Robert Muir added a comment -

          It is not the build process that requires simplification (is anybody really calling for that?), but rather developers' interaction with it?

          I am trying to simplify the build, with the prototype in SOLR-2002, whereas most things are in a single common-build and we invoke them recursively (things like rat, dist, test, javadocs).

          Right now it is complicated mostly because there are actually many build setups that are essentially standalone, and then have been linked to depend on each other.

          Show
          Robert Muir added a comment - It is not the build process that requires simplification (is anybody really calling for that?), but rather developers' interaction with it? I am trying to simplify the build, with the prototype in SOLR-2002 , whereas most things are in a single common-build and we invoke them recursively (things like rat, dist, test, javadocs). Right now it is complicated mostly because there are actually many build setups that are essentially standalone, and then have been linked to depend on each other.
          Hide
          Steve Rowe added a comment -

          It is not the build process that requires simplification (is anybody really calling for that?), but rather developers' interaction with it?

          I am trying to simplify the build, with the prototype in SOLR-2002, whereas most things are in a single common-build and we invoke them recursively (things like rat, dist, test, javadocs).

          I stand corrected. Build simplification is certainly a goal I support; without the flexibility afforded by a coherent build system, issues like this one would definitely be more difficult to implement.

          Show
          Steve Rowe added a comment - It is not the build process that requires simplification (is anybody really calling for that?), but rather developers' interaction with it? I am trying to simplify the build, with the prototype in SOLR-2002 , whereas most things are in a single common-build and we invoke them recursively (things like rat, dist, test, javadocs). I stand corrected. Build simplification is certainly a goal I support; without the flexibility afforded by a coherent build system, issues like this one would definitely be more difficult to implement.
          Hide
          Robert Muir added a comment -

          I stand corrected. Build simplification is certainly a goal I support; without the flexibility afforded by a coherent build system, issues like this one would definitely be more difficult to implement.

          Exactly, because if you were to write a patch for this issue today, you have to implement it in 4 or 5 different places at the moment (or at least employ dirty ant hacks)

          The same problem prevents us from really improving our build in other ways: for example we should probably support findbugs/pmd integration and things like that, and it should be simple.

          Show
          Robert Muir added a comment - I stand corrected. Build simplification is certainly a goal I support; without the flexibility afforded by a coherent build system, issues like this one would definitely be more difficult to implement. Exactly, because if you were to write a patch for this issue today, you have to implement it in 4 or 5 different places at the moment (or at least employ dirty ant hacks) The same problem prevents us from really improving our build in other ways: for example we should probably support findbugs/pmd integration and things like that, and it should be simple.
          Hide
          Chris Male added a comment -

          I don't see this issue as providing a complete Ant->Maven conversion system. The poms I envision will look just like the POM templates after they are interpolated - they will provide Maven coordinates for artifacts and their dependencies, nothing more. And the process to produce them can be carried out as part of generating Maven artifacts. I don't think they will need to be checked in, but they certainly could be.

          Its this sort of middle ground that I'm unsure about. We basically have maven support today but not really. Some people understand how it works, but I've certainly seen users who are dabbling in some development and are confused by our maven support. I cant think of another project that generates its poms (even ant itself has the poms well-formed inside its codebase) so suddenly our support has become more unusual.

          What I'm hoping is that we can actually add quality support for maven in such a way that users wouldn't be surprised, developers like you and I can get the benefits we know maven has, and that also simplifies what Robert has to do.

          If all we can do at the moment is bolt bits of maven onto ant, I'm unsure its worth while.

          Show
          Chris Male added a comment - I don't see this issue as providing a complete Ant->Maven conversion system. The poms I envision will look just like the POM templates after they are interpolated - they will provide Maven coordinates for artifacts and their dependencies, nothing more. And the process to produce them can be carried out as part of generating Maven artifacts. I don't think they will need to be checked in, but they certainly could be. Its this sort of middle ground that I'm unsure about. We basically have maven support today but not really. Some people understand how it works, but I've certainly seen users who are dabbling in some development and are confused by our maven support. I cant think of another project that generates its poms (even ant itself has the poms well-formed inside its codebase) so suddenly our support has become more unusual. What I'm hoping is that we can actually add quality support for maven in such a way that users wouldn't be surprised, developers like you and I can get the benefits we know maven has, and that also simplifies what Robert has to do. If all we can do at the moment is bolt bits of maven onto ant, I'm unsure its worth while.
          Hide
          Chris Male added a comment -

          I should add that I'm currently looking at how ant supports maven releases since it does have a contrib-like folder of poms, well formed.

          Show
          Chris Male added a comment - I should add that I'm currently looking at how ant supports maven releases since it does have a contrib-like folder of poms, well formed.
          Hide
          Steve Rowe added a comment -

          I have changed the issue summary and description to manual generation of authoritative POMs, rather than automatic synching of POM templates with Ant dependencies, which can be pursued later.

          Show
          Steve Rowe added a comment - I have changed the issue summary and description to manual generation of authoritative POMs, rather than automatic synching of POM templates with Ant dependencies, which can be pursued later.
          Hide
          Yonik Seeley added a comment -

          Full Maven POMs will include the information necessary to run a multi-module Maven build

          That sort of sounds like a parallel build process (i.e. you would be able to build lucene/solr itself with maven). Is it?
          We've avoided that type of thing in the past.

          Show
          Yonik Seeley added a comment - Full Maven POMs will include the information necessary to run a multi-module Maven build That sort of sounds like a parallel build process (i.e. you would be able to build lucene/solr itself with maven). Is it? We've avoided that type of thing in the past.
          Hide
          Steve Rowe added a comment -

          Full Maven POMs will include the information necessary to run a multi-module Maven build

          That sort of sounds like a parallel build process (i.e. you would be able to build lucene/solr itself with maven). Is it?
          We've avoided that type of thing in the past.

          Yes, it constitutes a parallel build process, but the Ant build process would remain the official build. For example, the artifacts produced by the Maven build will not be exactly the same as those produced by the Ant build process, and so cannot be used as release artifacts.

          It has been noted elsewhere that a Maven build has never been produced for Lucene. I hope in this issue to provide that, so that the discussion about whether or not to include it in the source tree has a concrete reference point.

          Show
          Steve Rowe added a comment - Full Maven POMs will include the information necessary to run a multi-module Maven build That sort of sounds like a parallel build process (i.e. you would be able to build lucene/solr itself with maven). Is it? We've avoided that type of thing in the past. Yes, it constitutes a parallel build process, but the Ant build process would remain the official build. For example, the artifacts produced by the Maven build will not be exactly the same as those produced by the Ant build process, and so cannot be used as release artifacts. It has been noted elsewhere that a Maven build has never been produced for Lucene. I hope in this issue to provide that, so that the discussion about whether or not to include it in the source tree has a concrete reference point.
          Hide
          Steve Rowe added a comment -

          First cut at Maven POMs for Lucene and Solr.

          The following script, which installs non-mavenized artifacts into the user's local Maven repository, must be run before the Maven build works:

           mvn install:install-file -DgroupId=com.ibm.icu -DartifactId=icu4j -Dversion=4.4 -Dpackaging=jar -Dfile=modules/analysis/icu/lib/icu4j-4_4.jar
           mvn install:install-file -DgroupId=xerces -DartifactId=xercesImpl -Dversion=2.9.1-patched-XERCESJ-1257 -Dpackaging=jar -Dfile=lucene/contrib/benchmark/lib/xerces-2.9.1-patched-XERCESJ-1257.jar
           mvn install:install-file -DgroupId=com.sleepycat -DartifactId=berkeleydb -Dversion=4.7.25 -Dpackaging=jar -Dfile=lucene/contrib/db/bdb/lib/db-4.7.25.jar
           mvn install:install-file -DgroupId=com.sleepycat -DartifactId=berkeleydb-je -Dversion=3.3.93 -Dpackaging=jar -Dfile=lucene/contrib/db/bdb-je/lib/je-3.3.93.jar
           mvn install:install-file -DgroupId=org.apache.solr -DartifactId=solr-commons-csv -Dversion=4.0-SNAPSHOT -Dpackaging=jar -Dfile=solr/lib/commons-csv-1.0-SNAPSHOT-r966014.jar
           mvn install:install-file -DgroupId=org.apache.solr -DartifactId=solr-noggit -Dversion=r944541 -Dpackaging=jar -Dfile=solr/lib/apache-solr-noggit-r944541.jar
           mvn install:install-file -DgroupId=org.apache.tika -DartifactId=tika-core -Dversion=0.8-SNAPSHOT -Dpackaging=jar -Dfile=solr/contrib/extraction/lib/tika-core-0.8-SNAPSHOT.jar
           jar xvf solr/contrib/extraction/lib/tika-parsers-0.8-SNAPSHOT.jar META-INF/maven/org.apache.tika/tika-parsers/pom.xml
           mvn install:install-file -Dfile=solr/contrib/extraction/lib/tika-parsers-0.8-SNAPSHOT.jar -DpomFile=META-INF/maven/org.apache.tika/tika-parsers/pom.xml
           rm -rf META-INF
           

          Everything compiles (using mvn -DskipTests clean install). Except for two Solr tests, all tests pass. The two tests that fail for me (on Win Vista 64, Sun JDK 1.6.0_13) are: JettyWebAppTest and Solr Cell's ExtractingRequestHandlerTest.testArabicPDF().

          I have added configurations for running appassembler-maven-plugin (to produce run-time windows and linux scripts with lib/ dirs containing dependencies) for all classes with main() functions that are intended as tools rather than static tests.

          I haven't tried using the POMs as project configuration for IntelliJ, so I'm not sure how that will go.

          This is a work-in-progress. Please test and let me know if you find problems, or would like to see more stuff added, or if you think the design is all wrong.

          One challenge worth noting is that Maven disallows aggregator POMs (those with a <modules> section directing nested Maven invocations) from producing any artifacts other than the POM itself. As a result, the 4 or 5 locations in the Lucene/Solr build that have "mixed content" (to borrow the XML term for mixed nested element and text content) have to be fudged; e.g. the Lucene and Solr core POMs are located in lucene/src/ and solr/src/, because the aggregator POMs at lucene/ and solr/ can't produce binary artifacts. (The conventional Maven way to address this kind of issue is to make the "core" modules be siblings of the nested modules.)

          Show
          Steve Rowe added a comment - First cut at Maven POMs for Lucene and Solr. The following script, which installs non-mavenized artifacts into the user's local Maven repository, must be run before the Maven build works: mvn install:install-file -DgroupId=com.ibm.icu -DartifactId=icu4j -Dversion=4.4 -Dpackaging=jar -Dfile=modules/analysis/icu/lib/icu4j-4_4.jar mvn install:install-file -DgroupId=xerces -DartifactId=xercesImpl -Dversion=2.9.1-patched-XERCESJ-1257 -Dpackaging=jar -Dfile=lucene/contrib/benchmark/lib/xerces-2.9.1-patched-XERCESJ-1257.jar mvn install:install-file -DgroupId=com.sleepycat -DartifactId=berkeleydb -Dversion=4.7.25 -Dpackaging=jar -Dfile=lucene/contrib/db/bdb/lib/db-4.7.25.jar mvn install:install-file -DgroupId=com.sleepycat -DartifactId=berkeleydb-je -Dversion=3.3.93 -Dpackaging=jar -Dfile=lucene/contrib/db/bdb-je/lib/je-3.3.93.jar mvn install:install-file -DgroupId=org.apache.solr -DartifactId=solr-commons-csv -Dversion=4.0-SNAPSHOT -Dpackaging=jar -Dfile=solr/lib/commons-csv-1.0-SNAPSHOT-r966014.jar mvn install:install-file -DgroupId=org.apache.solr -DartifactId=solr-noggit -Dversion=r944541 -Dpackaging=jar -Dfile=solr/lib/apache-solr-noggit-r944541.jar mvn install:install-file -DgroupId=org.apache.tika -DartifactId=tika-core -Dversion=0.8-SNAPSHOT -Dpackaging=jar -Dfile=solr/contrib/extraction/lib/tika-core-0.8-SNAPSHOT.jar jar xvf solr/contrib/extraction/lib/tika-parsers-0.8-SNAPSHOT.jar META-INF/maven/org.apache.tika/tika-parsers/pom.xml mvn install:install-file -Dfile=solr/contrib/extraction/lib/tika-parsers-0.8-SNAPSHOT.jar -DpomFile=META-INF/maven/org.apache.tika/tika-parsers/pom.xml rm -rf META-INF Everything compiles (using mvn -DskipTests clean install ). Except for two Solr tests, all tests pass. The two tests that fail for me (on Win Vista 64, Sun JDK 1.6.0_13) are: JettyWebAppTest and Solr Cell's ExtractingRequestHandlerTest.testArabicPDF(). I have added configurations for running appassembler-maven-plugin (to produce run-time windows and linux scripts with lib/ dirs containing dependencies) for all classes with main() functions that are intended as tools rather than static tests. I haven't tried using the POMs as project configuration for IntelliJ, so I'm not sure how that will go. This is a work-in-progress. Please test and let me know if you find problems, or would like to see more stuff added, or if you think the design is all wrong. One challenge worth noting is that Maven disallows aggregator POMs (those with a <modules> section directing nested Maven invocations) from producing any artifacts other than the POM itself. As a result, the 4 or 5 locations in the Lucene/Solr build that have "mixed content" (to borrow the XML term for mixed nested element and text content) have to be fudged; e.g. the Lucene and Solr core POMs are located in lucene/src/ and solr/src/ , because the aggregator POMs at lucene/ and solr/ can't produce binary artifacts. (The conventional Maven way to address this kind of issue is to make the "core" modules be siblings of the nested modules.)
          Hide
          Robert Muir added a comment -

          Except for two Solr tests, all tests pass. The two tests that fail for me (on Win Vista 64, Sun JDK 1.6.0_13) are: JettyWebAppTest and Solr Cell's ExtractingRequestHandlerTest.testArabicPDF().

          maybe it is an icu classpath issue. modules/analysis and solr/contrib/extraction both use different, out of date versions.
          i opened LUCENE-2683

          Show
          Robert Muir added a comment - Except for two Solr tests, all tests pass. The two tests that fail for me (on Win Vista 64, Sun JDK 1.6.0_13) are: JettyWebAppTest and Solr Cell's ExtractingRequestHandlerTest.testArabicPDF(). maybe it is an icu classpath issue. modules/analysis and solr/contrib/extraction both use different, out of date versions. i opened LUCENE-2683
          Hide
          Chris Male added a comment -

          Hi Steven,

          The following script, which installs non-mavenized artifacts into the user's local Maven repository

          Would system scope dependencies be an alternative way? Then we can specify the systemPath to where the jars are in the lib folders.
          Reason I suggest this is that if someone replaces one of those jar (which will happen I'm sure) or simply adds another, then we cannot
          simply update the appropriate pom. We'd also need to install the jars locally (possibly again).

          Show
          Chris Male added a comment - Hi Steven, The following script, which installs non-mavenized artifacts into the user's local Maven repository Would system scope dependencies be an alternative way? Then we can specify the systemPath to where the jars are in the lib folders. Reason I suggest this is that if someone replaces one of those jar (which will happen I'm sure) or simply adds another, then we cannot simply update the appropriate pom. We'd also need to install the jars locally (possibly again).
          Hide
          Steve Rowe added a comment -

          Would system scope dependencies be an alternative way?

          I thought of that, and it probably makes sense for most of them.

          But a jar like tika-parsers-0.8-SNAPSHOT.jar, with 10 or 15 dependencies, would be even messier to deal with as a system dependency than as a locally installed artifact. We would have to go look at the POM for it (contained conveniently within the jar, as it happens), compare the dependencies against the versions of the local dependencies (which would have previously had to be transferred to the both the Solr Cell and the DIH Extras POMs) and make changes in both places.

          Show
          Steve Rowe added a comment - Would system scope dependencies be an alternative way? I thought of that, and it probably makes sense for most of them. But a jar like tika-parsers-0.8-SNAPSHOT.jar, with 10 or 15 dependencies, would be even messier to deal with as a system dependency than as a locally installed artifact. We would have to go look at the POM for it (contained conveniently within the jar, as it happens), compare the dependencies against the versions of the local dependencies (which would have previously had to be transferred to the both the Solr Cell and the DIH Extras POMs) and make changes in both places.
          Hide
          Steve Rowe added a comment -

          This version of the patch puts the third-party non-mavenized .jar installation stuff into a profile in the top-level POM.

          So instead of running the above-listed script, instead just run the following Maven invocation:

          mvn -N -Pbootstrap install
          

          -N disables recursive building, -Pbootstrap activates the "bootstrap" profile (defined in the POM) and the local repo artifact installation stuff is bound to the "install" phase in the "bootstrap" profile.

          I discovered that the Tika artifact snapshots (tika-core and tika-parsers) are hosted at the Apache snapshots repository, and that was causing these dependencies not to work properly, so I switched their coordinates to org.apache.solr:solr-tika-*:4.0-SNAPSHOT, so that they will remain stable.

          Also, the previous patch had some weirdness with the Solr DIH Extras POM - it wasn't being treated as a new file, but rather a diff from some other file (the POM template, I think) - anyway, this patch fixes the problem.

          Everything still compiles, and all tests except the two previously failing tests still pass.

          Show
          Steve Rowe added a comment - This version of the patch puts the third-party non-mavenized .jar installation stuff into a profile in the top-level POM. So instead of running the above-listed script, instead just run the following Maven invocation: mvn -N -Pbootstrap install -N disables recursive building, -Pbootstrap activates the "bootstrap" profile (defined in the POM) and the local repo artifact installation stuff is bound to the "install" phase in the "bootstrap" profile. I discovered that the Tika artifact snapshots (tika-core and tika-parsers) are hosted at the Apache snapshots repository, and that was causing these dependencies not to work properly, so I switched their coordinates to org.apache.solr:solr-tika-*:4.0-SNAPSHOT, so that they will remain stable. Also, the previous patch had some weirdness with the Solr DIH Extras POM - it wasn't being treated as a new file, but rather a diff from some other file (the POM template, I think) - anyway, this patch fixes the problem. Everything still compiles, and all tests except the two previously failing tests still pass.
          Hide
          Steve Rowe added a comment -

          Except for two Solr tests, all tests pass. The two tests that fail for me (on Win Vista 64, Sun JDK 1.6.0_13) are: JettyWebAppTest and Solr Cell's ExtractingRequestHandlerTest.testArabicPDF().

          maybe it is an icu classpath issue. modules/analysis and solr/contrib/extraction both use different, out of date versions.
          i opened LUCENE-2683

          I dropped in the ICU4J 4.4.2 jar, and everything compiled and all tests succeeded under Maven except for the same two tests, which continued to fail.

          Show
          Steve Rowe added a comment - Except for two Solr tests, all tests pass. The two tests that fail for me (on Win Vista 64, Sun JDK 1.6.0_13) are: JettyWebAppTest and Solr Cell's ExtractingRequestHandlerTest.testArabicPDF(). maybe it is an icu classpath issue. modules/analysis and solr/contrib/extraction both use different, out of date versions. i opened LUCENE-2683 I dropped in the ICU4J 4.4.2 jar, and everything compiled and all tests succeeded under Maven except for the same two tests, which continued to fail.
          Hide
          Robert Muir added a comment -

          I dropped in the ICU4J 4.4.2 jar, and everything compiled and all tests succeeded under Maven except for the same two tests, which continued to fail.

          Well it wasn't the icu issue then, but we needed to upgrade anyway for the bugfixes, no loss.

          What failure are you seeing? When prototyping the the ant system refactor (SOLR-2002 / SOLR-2002_merged.patch), i had a few issues
          with some tests but I don't remember the details. I remember it was really important to get the CWD of the junit run correct, e.g. in that patch its:

          <property name="tests.userdir" location="src/test/resources"/>
            and later invoked as junit's dir="${tests.userdir}" attribute
          
          Show
          Robert Muir added a comment - I dropped in the ICU4J 4.4.2 jar, and everything compiled and all tests succeeded under Maven except for the same two tests, which continued to fail. Well it wasn't the icu issue then, but we needed to upgrade anyway for the bugfixes, no loss. What failure are you seeing? When prototyping the the ant system refactor ( SOLR-2002 / SOLR-2002 _merged.patch), i had a few issues with some tests but I don't remember the details. I remember it was really important to get the CWD of the junit run correct, e.g. in that patch its: <property name="tests.userdir" location="src/test/resources"/> and later invoked as junit's dir="${tests.userdir}" attribute
          Hide
          Steve Rowe added a comment -

          What failure are you seeing?

          Two apparently unrelated errors:

          1. JettyWebappTest.testJSP() returns an HTTP 500 error as soon as it attempts to connect to the Jetty Server instantiated in setUp(), on line #88. I haven't figured out how to set Jetty Logging up so that I have visibility on what's happening yet. I see a call setting the solr.solr.home property in setUp() on line #49, so maybe that value is not in sync with the current directory I transferred from the Ant configuration.
          2. solr/contrib/extraction's ExtractingRequestHandlerTest.testArabicPDF() test fails because no result is found containing the extracted Arabic term (line #353). I have traced content extraction up through plain text extraction from the PDF, and that seems to be working, so I don't think it's a classpath issue with Tika or PDFBox. I need to keep tracing to see happens after the text extraction.
          Show
          Steve Rowe added a comment - What failure are you seeing? Two apparently unrelated errors: JettyWebappTest.testJSP() returns an HTTP 500 error as soon as it attempts to connect to the Jetty Server instantiated in setUp() , on line #88. I haven't figured out how to set Jetty Logging up so that I have visibility on what's happening yet. I see a call setting the solr.solr.home property in setUp() on line #49, so maybe that value is not in sync with the current directory I transferred from the Ant configuration. solr/contrib/extraction 's ExtractingRequestHandlerTest.testArabicPDF() test fails because no result is found containing the extracted Arabic term (line #353). I have traced content extraction up through plain text extraction from the PDF, and that seems to be working, so I don't think it's a classpath issue with Tika or PDFBox. I need to keep tracing to see happens after the text extraction.
          Hide
          Steve Rowe added a comment -

          Both of the failing tests were caused by missing dependencies: jetty jsp-2.1 (scope=provided) was missing from solr/src/pom.xml, and icu4j was missing from solr/contrib/extraction/pom.xml.

          With this version of the patch, all tests now pass.

          IntelliJ IDEA project import from the Maven configuration mostly works, with the exception that because the Solr core and Solrj modules each include two source directories, and IntelliJ doesn't grok this fact from the POMs (specified via build-helper-maven-plugin configuration), builds including these two modules fail, since they aren't seeing all of their source code.

          http://youtrack.jetbrains.net/issue/IDEA-27176 has a work-around for this issue: click on "update folders for all projects" button in the maven projects pane - this works for Solr core (webapp/src/ is added a source root), but not for Solrj, likely because the second source root solr/src/common/ is a sibling of the module root solr/src/solrj/, rather than a child.

          Show
          Steve Rowe added a comment - Both of the failing tests were caused by missing dependencies: jetty jsp-2.1 (scope=provided) was missing from solr/src/pom.xml , and icu4j was missing from solr/contrib/extraction/pom.xml . With this version of the patch, all tests now pass. IntelliJ IDEA project import from the Maven configuration mostly works, with the exception that because the Solr core and Solrj modules each include two source directories, and IntelliJ doesn't grok this fact from the POMs (specified via build-helper-maven-plugin configuration), builds including these two modules fail, since they aren't seeing all of their source code. http://youtrack.jetbrains.net/issue/IDEA-27176 has a work-around for this issue: click on "update folders for all projects" button in the maven projects pane - this works for Solr core ( webapp/src/ is added a source root), but not for Solrj, likely because the second source root solr/src/common/ is a sibling of the module root solr/src/solrj/ , rather than a child.
          Hide
          Steve Rowe added a comment -

          icu4j was missing from solr/contrib/extraction/pom.xml

          See SOLR-1813 for details.

          Show
          Steve Rowe added a comment - icu4j was missing from solr/contrib/extraction/pom.xml See SOLR-1813 for details.
          Hide
          Steve Rowe added a comment -

          Synch'd to trunk (ICU4J and Xerces-J dependency upgrades). Use mvn -N -Pbootstrap install from the top-level directory in order to install the new dependencies.

          Show
          Steve Rowe added a comment - Synch'd to trunk (ICU4J and Xerces-J dependency upgrades). Use mvn -N -Pbootstrap install from the top-level directory in order to install the new dependencies.
          Hide
          Steve Rowe added a comment -

          Updated to trunk, including addition of a POM for solr/contrib/analysis-extras/ and upgraded dependencies.

          All tests pass.

          Show
          Steve Rowe added a comment - Updated to trunk, including addition of a POM for solr/contrib/analysis-extras/ and upgraded dependencies. All tests pass.
          Hide
          Steve Rowe added a comment -

          Earlier Maven work

          Show
          Steve Rowe added a comment - Earlier Maven work
          Hide
          Steve Rowe added a comment - - edited

          Changes in this version of the patch:

          1. Reduced configuration redundancy: as much as possible, shared configuration is now located in the lucene-parent POM.
          2. The solr-aggregator POM is converted to a solr-parent POM, and all Solr modules are now children of it rather than of lucene-parent. This POM hosts Solr-specific shared configuration.
          3. The solr war is now generated in a new POM located in solr/src/webapp/. This POM also includes an untested maven-jetty-plugin configuration, for use in starting up Solr from Maven, as requested in SOLR-1218.
          4. A profile in the lucene-parent POM to generate source jars for each module: source-jar.
          5. A profile in the lucene-parent POM to generate javadoc jars for each module: javadocs-jar.
          6. Consistent MANIFEST.MF entries in all archives, including timestamps generated using embedded Groovy via gmaven-plugin and the Subversion revision from buildhelper-maven-plugin.
          7. NOTICE.txt and LICENSE.txt are now placed in META-INF/ in all archives
          8. The Maven plugin versions are updated to the latest available.

          All tests pass.

          If you have used previous versions of the patch, you should re-run mvn -N -P bootstrap install from the top level, to install the non-maven dependencies in your local Maven repository. (Some of the jar names have changed to better reflect their names as used in Solr.)

          To create binary, javadoc and source jars, and install them in your local repository, run the following at any level:

          mvn -DskipTests -P javadocs-jar,source-jar clean install
          
          Show
          Steve Rowe added a comment - - edited Changes in this version of the patch: Reduced configuration redundancy: as much as possible, shared configuration is now located in the lucene-parent POM. The solr-aggregator POM is converted to a solr-parent POM, and all Solr modules are now children of it rather than of lucene-parent . This POM hosts Solr-specific shared configuration. The solr war is now generated in a new POM located in solr/src/webapp/ . This POM also includes an untested maven-jetty-plugin configuration, for use in starting up Solr from Maven, as requested in SOLR-1218 . A profile in the lucene-parent POM to generate source jars for each module: source-jar . A profile in the lucene-parent POM to generate javadoc jars for each module: javadocs-jar . Consistent MANIFEST.MF entries in all archives, including timestamps generated using embedded Groovy via gmaven-plugin and the Subversion revision from buildhelper-maven-plugin . NOTICE.txt and LICENSE.txt are now placed in META-INF/ in all archives The Maven plugin versions are updated to the latest available. All tests pass. If you have used previous versions of the patch, you should re-run mvn -N -P bootstrap install from the top level, to install the non-maven dependencies in your local Maven repository. (Some of the jar names have changed to better reflect their names as used in Solr.) To create binary, javadoc and source jars, and install them in your local repository, run the following at any level: mvn -DskipTests -P javadocs-jar,source-jar clean install
          Hide
          Steve Rowe added a comment -

          All tests pass again with this patch. Solr test resource structrual changes from SOLR-2299 (r1054169) are now accounted for. All test CWDs are set to {{$

          {project.build.testOutputDirectory}

          }}, so that any stray files put on the CWD by tests will be cleaned up using either mvn clean or ant clean.

          Show
          Steve Rowe added a comment - All tests pass again with this patch. Solr test resource structrual changes from SOLR-2299 (r1054169) are now accounted for. All test CWDs are set to {{$ {project.build.testOutputDirectory} }}, so that any stray files put on the CWD by tests will be cleaned up using either mvn clean or ant clean .
          Hide
          Steve Rowe added a comment -

          Resync with trunk to move lucene/contrib/benchmark/ to modules/benchmark/: LUCENE-2845

          Show
          Steve Rowe added a comment - Resync with trunk to move lucene/contrib/benchmark/ to modules/benchmark/ : LUCENE-2845
          Hide
          Steve Rowe added a comment -

          Removed lucene/contrib/remote/ (LUCENE-2837)

          Show
          Steve Rowe added a comment - Removed lucene/contrib/remote/ ( LUCENE-2837 )
          Hide
          Steve Rowe added a comment - - edited

          Added profiles to populate internal repositories at lucene/dist/maven/ and solr/dist/maven/ with generated artifacts.

          To populate lucene/dist/maven/ with POMs and binary, source and javadoc artifacts, run the following from the top level Lucene/Solr directory:

          mvn -N -P bootstrap,deploy-to-lucene-dist-maven-repository deploy
          cd lucene
          mvn -DskipTests -Pdeploy-to-lucene-dist-maven-repository,javadocs-jar,source-jar deploy
          cd ../modules
          mvn -DskipTests -Pdeploy-to-lucene-dist-maven-repository,javadocs-jar,source-jar deploy
          

          To populate solr/dist/maven/, run the following from the top level Lucene/Solr directory:

          mvn -N -P bootstrap install
          cd solr
          mvn -DskipTests -Pdeploy-to-solr-dist-maven-repository,javadocs-jar,source-jar deploy
          
          Show
          Steve Rowe added a comment - - edited Added profiles to populate internal repositories at lucene/dist/maven/ and solr/dist/maven/ with generated artifacts. To populate lucene/dist/maven/ with POMs and binary, source and javadoc artifacts, run the following from the top level Lucene/Solr directory: mvn -N -P bootstrap,deploy-to-lucene-dist-maven-repository deploy cd lucene mvn -DskipTests -Pdeploy-to-lucene-dist-maven-repository,javadocs-jar,source-jar deploy cd ../modules mvn -DskipTests -Pdeploy-to-lucene-dist-maven-repository,javadocs-jar,source-jar deploy To populate solr/dist/maven/ , run the following from the top level Lucene/Solr directory: mvn -N -P bootstrap install cd solr mvn -DskipTests -Pdeploy-to-solr-dist-maven-repository,javadocs-jar,source-jar deploy
          Hide
          Steve Rowe added a comment -

          carrot2 dependency upgraded to 3.4.2: SOLR-2296

          Show
          Steve Rowe added a comment - carrot2 dependency upgraded to 3.4.2: SOLR-2296
          Hide
          Ryan McKinley added a comment -

          This is looking good!

          IIUC, this will be a parallel build system to ant. The build and test is independent of anything the ant build does.

          If we take this route, we should probably drop the -pom.xml.templates and the and --generate-maven-artifacts target.

          Show
          Ryan McKinley added a comment - This is looking good! IIUC, this will be a parallel build system to ant. The build and test is independent of anything the ant build does. If we take this route, we should probably drop the -pom.xml.templates and the and --generate-maven-artifacts target.
          Hide
          Steve Rowe added a comment -

          IIUC, this will be a parallel build system to ant. The build and test is independent of anything the ant build does.

          Yes, except that the two systems share build output directories.

          If we take this route, we should probably drop the -pom.xml.templates and the and --generate-maven-artifacts target.

          I agree.

          The -pom.xml.templates have never been fully correct (e.g. missing dependencies) and are unmaintained.

          I'm working on replacing generate-maven-artifacts.

          Show
          Steve Rowe added a comment - IIUC, this will be a parallel build system to ant. The build and test is independent of anything the ant build does. Yes, except that the two systems share build output directories. If we take this route, we should probably drop the -pom.xml.templates and the and --generate-maven-artifacts target. I agree. The -pom.xml.templates have never been fully correct (e.g. missing dependencies) and are unmaintained. I'm working on replacing generate-maven-artifacts.
          Hide
          Steve Rowe added a comment - - edited
          • Set snapshot updatePolicy to never for both the apache.snapshots and the carrot2.org Maven repositories, so that they won't constantly be checked for snapshot updates.
          • Consolidated distribution-related profiles to just one named dist
          • Solr-specific noggit and commons-csv jars are now properly placed in solr/dist/maven/ when deploying with the dist profile
          • No longer setting the repositories' uniqueVersion to false when deploying under the dist profile; as a result, snapshot artifacts' names will include build timestamps instead of "SNAPSHOT" in */dist/maven/.
          • mvn clean from lucene/src/ and solr/src/ now removes lucene/dist/ and solr/dist/, respectively, in addition to the build directories already being removed.

          To populate both lucene/dist/maven/ and solr/dist/maven/, run from the top level:

          mvn -Pdist -DskipTests deploy
          

          To populate only lucene/dist/maven/, run from the top level:

          mvn -N -Pdist deploy
          cd lucene
          mvn -Pdist -DskipTests deploy
          cd ../modules
          mvn -Pdist -DskipTests deploy
          
          Show
          Steve Rowe added a comment - - edited Set snapshot updatePolicy to never for both the apache.snapshots and the carrot2.org Maven repositories, so that they won't constantly be checked for snapshot updates. Consolidated distribution-related profiles to just one named dist Solr-specific noggit and commons-csv jars are now properly placed in solr/dist/maven/ when deploying with the dist profile No longer setting the repositories' uniqueVersion to false when deploying under the dist profile; as a result, snapshot artifacts' names will include build timestamps instead of " SNAPSHOT " in */dist/maven/ . mvn clean from lucene/src/ and solr/src/ now removes lucene/dist/ and solr/dist/ , respectively, in addition to the build directories already being removed. To populate both lucene/dist/maven/ and solr/dist/maven/ , run from the top level: mvn -Pdist -DskipTests deploy To populate only lucene/dist/maven/ , run from the top level: mvn -N -Pdist deploy cd lucene mvn -Pdist -DskipTests deploy cd ../modules mvn -Pdist -DskipTests deploy
          Hide
          Steve Rowe added a comment -

          This patch cuts over ant generate-maven-artifacts to directly use mvn. After applying this patch, Maven 2.2.1 (and maybe 3.0.X? - untested) must be installed on your machine in order to run ant generate-maven-artifacts.

          Other changes in this patch:

          1. Dropped all use of Maven Ant Tasks.
          2. The top-level ant generate-maven-artifacts now works and is the best way to perform this task, since it will create a single timestamp for all artifacts; this target can also be run from either solr/ or lucene/.
          3. Removed the generate-maven-artifacts target from modules/build.xml, and transferred the responsibility for generating modules/* maven artifacts to lucene/build.xml.
          4. The solr/src/webapp/ module no longer installs or deploys its (empty) sources jar.
          5. Remote Maven artifact deployment is no longer included in the Ant build - this can be performed by the Maven build.
          6. mvn clean no longer removes solr/dist/ or lucene/dist/, for two reasons:
            1. The Ant build populates dist/ with things that the Maven build should not remove. Removing just dist/maven/ won't work, because:
            2. I couldn't find a nice/simple way to remove a directory just once in the reactor build. The previous patch attempted to do this from the lucene core and solr core modules, but that solution was deleting their deployed parent POMs, since the reactor build orders the Solr and Lucene parent POMs before the core modules (the parent relationship requires this).

          Left to do:

          1. Add Ant targets to test the Maven artifacts
          2. Backport to branch_3x
          Show
          Steve Rowe added a comment - This patch cuts over ant generate-maven-artifacts to directly use mvn . After applying this patch, Maven 2.2.1 (and maybe 3.0.X? - untested) must be installed on your machine in order to run ant generate-maven-artifacts . Other changes in this patch: Dropped all use of Maven Ant Tasks. The top-level ant generate-maven-artifacts now works and is the best way to perform this task, since it will create a single timestamp for all artifacts; this target can also be run from either solr/ or lucene/ . Removed the generate-maven-artifacts target from modules/build.xml , and transferred the responsibility for generating modules/* maven artifacts to lucene/build.xml . The solr/src/webapp/ module no longer installs or deploys its (empty) sources jar. Remote Maven artifact deployment is no longer included in the Ant build - this can be performed by the Maven build. mvn clean no longer removes solr/dist/ or lucene/dist/ , for two reasons: The Ant build populates dist/ with things that the Maven build should not remove. Removing just dist/maven/ won't work, because: I couldn't find a nice/simple way to remove a directory just once in the reactor build. The previous patch attempted to do this from the lucene core and solr core modules, but that solution was deleting their deployed parent POMs, since the reactor build orders the Solr and Lucene parent POMs before the core modules (the parent relationship requires this). Left to do: Add Ant targets to test the Maven artifacts Backport to branch_3x
          Hide
          Chris A. Mattmann added a comment -

          Hey Guys,

          I've set this up on some other Apache projects (Nutch, Tika [NetCDF4] and SIS so far), and basically it involved:

          1. moddin'g build.xml according to Sonatype's guide (see build.xml section)

          https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide

          2. adding pom.xmls for each artifact to be published

          I'll throw together a patch for this and see if I can't make this process a bit easier.

          Thanks.

          Cheers,
          Chris

          Show
          Chris A. Mattmann added a comment - Hey Guys, I've set this up on some other Apache projects (Nutch, Tika [NetCDF4] and SIS so far), and basically it involved: 1. moddin'g build.xml according to Sonatype's guide (see build.xml section) https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide 2. adding pom.xmls for each artifact to be published I'll throw together a patch for this and see if I can't make this process a bit easier. Thanks. Cheers, Chris
          Hide
          Steve Rowe added a comment -

          In this patch:

          1. ant generate-maven-artifacts now works the same as it does on trunk without this patch – using maven-ant-tasks – except that instead of using the POM templates, the POMs provided in the patch are used.
          2. ant generate-maven-artifacts now functions properly at the top level, from lucene/, from modules/, and from solr/.
          3. Removed all -source.jar and -javadoc.jar generation related functionality from the POMs, as well as the dist profile - the Ant build is responsible for putting together the maven artifacts.
          4. Removed the POM templates, except for the two required to deploy the solr-noggit and solr-commons-csv artifacts from the Ant build.
          5. Modified the Maven artifact handling in the Ant build, including artifact signing, to be correct.
          6. Based on feedback from Stevo Slavić <http://www.mail-archive.com/solr-user@lucene.apache.org/msg45656.html>, added explicit <groupId>'s to the POMs that didn't have them, and added explicit <relativePath>'s to the <parent> declarations in all POMs.

          I think this patch is ready to be committed to trunk.

          I'll post a branch_3x version of this patch tomorrow, and then I think the patches on this issue will be complete.

          Show
          Steve Rowe added a comment - In this patch: ant generate-maven-artifacts now works the same as it does on trunk without this patch – using maven-ant-tasks – except that instead of using the POM templates, the POMs provided in the patch are used. ant generate-maven-artifacts now functions properly at the top level, from lucene/ , from modules/ , and from solr/ . Removed all -source.jar and -javadoc.jar generation related functionality from the POMs, as well as the dist profile - the Ant build is responsible for putting together the maven artifacts. Removed the POM templates, except for the two required to deploy the solr-noggit and solr-commons-csv artifacts from the Ant build. Modified the Maven artifact handling in the Ant build, including artifact signing, to be correct. Based on feedback from Stevo Slavić < http://www.mail-archive.com/solr-user@lucene.apache.org/msg45656.html >, added explicit <groupId> 's to the POMs that didn't have them, and added explicit <relativePath> 's to the <parent> declarations in all POMs. I think this patch is ready to be committed to trunk. I'll post a branch_3x version of this patch tomorrow, and then I think the patches on this issue will be complete.
          Hide
          Robert Muir added a comment -

          I think this patch is ready to be committed to trunk.

          Well first of all, you obviously worked hard on this, but we need to think this one through before committing.
          Can we put this code in a separate project, that takes care of maven support for lucene?
          The problem is there are two camps "die maven die" and "maven or die". There will never be consensus.

          The only way for maven to survive, is for the users that care about it, to support itself, just like other packaging systems
          such as debian, redhat rpm, freebsd/mac ports, etc etc that we lucene, don't deal with.
          They can't continue to whine to people like me, who don't give a shit about it, to support it and produce its crazy ass
          complicated artifacts.

          Instead the people who care about these packaging systems, and know how to make them work must deal with them.

          Personally I really don't like:

          • Having two build systems
          • Having one build system (ant) rely upon the other (maven) to create release artifacts.

          Basically, the ant build system is our build. I think it needs to be able to fully build lucene for a
          release without involving any other build systems such as Make or Maven.

          Show
          Robert Muir added a comment - I think this patch is ready to be committed to trunk. Well first of all, you obviously worked hard on this, but we need to think this one through before committing. Can we put this code in a separate project, that takes care of maven support for lucene? The problem is there are two camps "die maven die" and "maven or die". There will never be consensus. The only way for maven to survive, is for the users that care about it, to support itself, just like other packaging systems such as debian, redhat rpm, freebsd/mac ports, etc etc that we lucene, don't deal with. They can't continue to whine to people like me, who don't give a shit about it, to support it and produce its crazy ass complicated artifacts. Instead the people who care about these packaging systems, and know how to make them work must deal with them. Personally I really don't like: Having two build systems Having one build system (ant) rely upon the other (maven) to create release artifacts. Basically, the ant build system is our build. I think it needs to be able to fully build lucene for a release without involving any other build systems such as Make or Maven.
          Hide
          Steve Rowe added a comment -

          Can we put this code in a separate project, that takes care of maven support for lucene?

          I'd rather not. The Lucene project has published Maven artifacts since the 1.9.1 release. I think we should continue to do that.

          The only way for maven to survive, is for the users that care about it, to support itself, just like other packaging systems such as debian, redhat rpm, freebsd/mac ports, etc etc that we lucene, don't deal with.

          OK, those are pretty obviously red herrings. Can we concentrate on the actual issue here without dragging in those extraneous things? Maven artifacts, not those other things, have been provided by Lucene since the 1.9.1 release. We obviously do deal with Maven.

          They can't continue to whine to people like me, who don't give a shit about it, to support it and produce its crazy ass complicated artifacts.

          The latest patch on this release uses the Ant artifacts directly. POMs are provided. You know, just like it has been since the 1.9.1 release.

          Instead the people who care about these packaging systems, and know how to make them work must deal with them.

          Um, like the patch on this issue is doing?

          Basically, the ant build system is our build. I think it needs to be able to fully build lucene for a release without involving any other build systems such as Make or Maven.

          This patch uses the Ant-produced artifacts to prepare for Maven artifact publishing. Maven itself is not invoked in the process. An Ant plugin handles the artifact deployment.

          I seriously do not understand why this is such a big deal. Why can't we just keep publishing Maven artifacts? You know, like we have for the past 15-20 releases.

          Show
          Steve Rowe added a comment - Can we put this code in a separate project, that takes care of maven support for lucene? I'd rather not. The Lucene project has published Maven artifacts since the 1.9.1 release. I think we should continue to do that. The only way for maven to survive, is for the users that care about it, to support itself, just like other packaging systems such as debian, redhat rpm, freebsd/mac ports, etc etc that we lucene, don't deal with. OK, those are pretty obviously red herrings. Can we concentrate on the actual issue here without dragging in those extraneous things? Maven artifacts, not those other things, have been provided by Lucene since the 1.9.1 release. We obviously do deal with Maven. They can't continue to whine to people like me, who don't give a shit about it, to support it and produce its crazy ass complicated artifacts. The latest patch on this release uses the Ant artifacts directly. POMs are provided. You know, just like it has been since the 1.9.1 release. Instead the people who care about these packaging systems, and know how to make them work must deal with them. Um, like the patch on this issue is doing? Basically, the ant build system is our build. I think it needs to be able to fully build lucene for a release without involving any other build systems such as Make or Maven. This patch uses the Ant-produced artifacts to prepare for Maven artifact publishing. Maven itself is not invoked in the process. An Ant plugin handles the artifact deployment. I seriously do not understand why this is such a big deal. Why can't we just keep publishing Maven artifacts? You know, like we have for the past 15-20 releases.
          Hide
          Robert Muir added a comment -

          I'd rather not. The Lucene project has published Maven artifacts since the 1.9.1 release. I think we should continue to do that.

          Well I disagree, and I definitely don't think we should have 2 build systems.

          As I stated before, I don't think we are going to see consensus on this issue, because maven is too "imposing" on people
          who don't care about it... it seems maven users are only happy if every project they depend on is itself a maven project.

          Show
          Robert Muir added a comment - I'd rather not. The Lucene project has published Maven artifacts since the 1.9.1 release. I think we should continue to do that. Well I disagree, and I definitely don't think we should have 2 build systems. As I stated before, I don't think we are going to see consensus on this issue, because maven is too "imposing" on people who don't care about it... it seems maven users are only happy if every project they depend on is itself a maven project.
          Hide
          Chris Male added a comment -

          I'm a little lost at what this patch introduces that is imposing? Ant itself has maven support as part of its trunk code base so its clearly not too imposing for them.

          Is your issue that this patch introduces things that get in your way somehow with using ant to do builds? or are you against committing this due to your general concerns with Maven?

          Show
          Chris Male added a comment - I'm a little lost at what this patch introduces that is imposing? Ant itself has maven support as part of its trunk code base so its clearly not too imposing for them. Is your issue that this patch introduces things that get in your way somehow with using ant to do builds? or are you against committing this due to your general concerns with Maven?
          Hide
          Robert Muir added a comment -

          Is your issue that this patch introduces things that get in your way somehow with using ant to do builds? or are you against committing this due to your general concerns with Maven?

          Personally I don't care for maven, but I would be bringing up these same objections if we proposed
          building Redhat RPMs or FreeBSD ports. I would say, its best for us to stay out of the business
          of building packages for various systems, instead let the people that work with those packaging
          systems do it themselves, they will do a better job and supporting N build/packaging systems is
          too expensive for us. It makes releasing more complicated to deploy these special packages
          to the various repositories where they want to be, it forces the release manager to know about
          N packaging systems instead of concentrating on releasing the source code to an open
          source search engine library, which is what we should be doing.

          Show
          Robert Muir added a comment - Is your issue that this patch introduces things that get in your way somehow with using ant to do builds? or are you against committing this due to your general concerns with Maven? Personally I don't care for maven, but I would be bringing up these same objections if we proposed building Redhat RPMs or FreeBSD ports. I would say, its best for us to stay out of the business of building packages for various systems, instead let the people that work with those packaging systems do it themselves, they will do a better job and supporting N build/packaging systems is too expensive for us. It makes releasing more complicated to deploy these special packages to the various repositories where they want to be, it forces the release manager to know about N packaging systems instead of concentrating on releasing the source code to an open source search engine library, which is what we should be doing.
          Hide
          Chris Male added a comment -

          Alright I can appreciate your concern. I think comparing Maven to RPM or FreeBSD ports is going a little far, but I can understand the point you're making.

          What if this were committed so that those of us who do understand maven and do like using it, could?
          This issue about whether maven artifacts need to then be released or not can be part of a greater discussion (as is already taking place).

          By committing this we then make it easier for someone else outside of the project to create the correct artifacts which are then available from the
          central maven repository, if thats the decision thats made which is also the one you support.

          Show
          Chris Male added a comment - Alright I can appreciate your concern. I think comparing Maven to RPM or FreeBSD ports is going a little far, but I can understand the point you're making. What if this were committed so that those of us who do understand maven and do like using it, could? This issue about whether maven artifacts need to then be released or not can be part of a greater discussion (as is already taking place). By committing this we then make it easier for someone else outside of the project to create the correct artifacts which are then available from the central maven repository, if thats the decision thats made which is also the one you support.
          Hide
          Robert Muir added a comment -

          What if this were committed so that those of us who do understand maven and do like using it, could?
          This issue about whether maven artifacts need to then be released or not can be part of a greater discussion (as is already taking place).

          By committing this we then make it easier for someone else outside of the project to create the correct artifacts which are then available from the
          central maven repository, if thats the decision thats made which is also the one you support.

          Why does it need to be committed to our trunk? why can't it be on e.g. apache-extras and do all of this.

          This way, we can continue to evolve lucene (e.g. merge lucene and solr better), maintaining only ONE build system,
          and hopefully! release more often... if downstream maven support falls behind, it falls behind, but it shouldn't
          slow us down.

          Show
          Robert Muir added a comment - What if this were committed so that those of us who do understand maven and do like using it, could? This issue about whether maven artifacts need to then be released or not can be part of a greater discussion (as is already taking place). By committing this we then make it easier for someone else outside of the project to create the correct artifacts which are then available from the central maven repository, if thats the decision thats made which is also the one you support. Why does it need to be committed to our trunk? why can't it be on e.g. apache-extras and do all of this. This way, we can continue to evolve lucene (e.g. merge lucene and solr better), maintaining only ONE build system, and hopefully! release more often... if downstream maven support falls behind, it falls behind, but it shouldn't slow us down.
          Hide
          Chris Male added a comment -

          I can appreciate the need to keep our trunk lean and mean and I hope that your assertion that that'll mean more frequent releases holds true.
          I hope we also shine the light on other code in the trunk to see if it belongs there or could be spun off.

          I do continue to support the committing of this code but as you rightly pointed out, there won't be any consensus on the issue so I'll leave it at that.

          Show
          Chris Male added a comment - I can appreciate the need to keep our trunk lean and mean and I hope that your assertion that that'll mean more frequent releases holds true. I hope we also shine the light on other code in the trunk to see if it belongs there or could be spun off. I do continue to support the committing of this code but as you rightly pointed out, there won't be any consensus on the issue so I'll leave it at that.
          Hide
          Earwin Burrfoot added a comment -

          <irony>
          I am amazed at how idea/eclipse ant tasks managed to get into trunk.
          No, really, emacs is just enough for Lucene development, as proven by one of our most seasoned developers.
          So why the hell are those extras polluting main tree?
          </irony>

          Show
          Earwin Burrfoot added a comment - <irony> I am amazed at how idea/eclipse ant tasks managed to get into trunk. No, really, emacs is just enough for Lucene development, as proven by one of our most seasoned developers. So why the hell are those extras polluting main tree? </irony>
          Hide
          Robert Muir added a comment -

          I am amazed at how idea/eclipse ant tasks managed to get into trunk.

          I already stated on the "maven thread", that I have no problem with something like this patch being committed to dev-tools/
          alongside the eclipse and idea support... some others might object, but personally I see no problem.

          we already agreed these folders are simply developer tools, and if they start to rot, we will just delete them.

          my problem is with the release process and artifacts... so if we do this we need to be very clear that we are
          agreeing to put the maven support in dev-tools/ as "developer-support", but its not a build system, and it has
          no effect on artifacts... we don't release dev-tools, its just checked into svn to help developers.

          Show
          Robert Muir added a comment - I am amazed at how idea/eclipse ant tasks managed to get into trunk. I already stated on the "maven thread", that I have no problem with something like this patch being committed to dev-tools/ alongside the eclipse and idea support... some others might object, but personally I see no problem. we already agreed these folders are simply developer tools, and if they start to rot, we will just delete them. my problem is with the release process and artifacts... so if we do this we need to be very clear that we are agreeing to put the maven support in dev-tools/ as "developer-support", but its not a build system, and it has no effect on artifacts... we don't release dev-tools, its just checked into svn to help developers.
          Hide
          Chris Male added a comment -

          That was basically what I was getting at (perhaps not clearly enough).

          Would a satisfactory compromise be to view this patch as adding development support for maven, which is not to do with whether maven artifacts are released or not?

          The discussion about release process, artifacts and build system flamewars can then happen outside of this.

          Show
          Chris Male added a comment - That was basically what I was getting at (perhaps not clearly enough). Would a satisfactory compromise be to view this patch as adding development support for maven, which is not to do with whether maven artifacts are released or not? The discussion about release process, artifacts and build system flamewars can then happen outside of this.
          Hide
          Earwin Burrfoot added a comment -

          we need to be very clear .... and it has no effect on artifacts

          I feel something was missed in the heat of debate. Eg:

          The latest patch on this release uses the Ant artifacts directly.

          This patch uses the Ant-produced artifacts to prepare for Maven artifact publishing.

          Maven itself is not invoked in the process. An Ant plugin handles the artifact deployment.

          I will now try to decipher these quotes.
          It seems the patch takes the artifacts produced by Ant, as a part of our usual (and only) build process, and shoves it down Maven repository's throat along with a bunch of pom-descriptors.
          Nothing else is happening.

          Also, after everything that has been said, I think nobody in his right mind will force anyone to actually use the Ant target in question as a part of release. But it's nice to have it around, in case some user-friendly commiter would like to push (I'd like to reiterate - ant generated) artifacts into Maven.

          Show
          Earwin Burrfoot added a comment - we need to be very clear .... and it has no effect on artifacts I feel something was missed in the heat of debate. Eg: The latest patch on this release uses the Ant artifacts directly. This patch uses the Ant-produced artifacts to prepare for Maven artifact publishing. Maven itself is not invoked in the process. An Ant plugin handles the artifact deployment. I will now try to decipher these quotes. It seems the patch takes the artifacts produced by Ant, as a part of our usual (and only) build process, and shoves it down Maven repository's throat along with a bunch of pom-descriptors. Nothing else is happening. Also, after everything that has been said, I think nobody in his right mind will force anyone to actually use the Ant target in question as a part of release. But it's nice to have it around, in case some user-friendly commiter would like to push (I'd like to reiterate - ant generated) artifacts into Maven.
          Hide
          Robert Muir added a comment -

          Chris: well thats the problem with maven, it tries to be too many things, a dependency management tool,
          a packaging system, a build system, ...

          So, thats why I said we have to just be very clear about which exact scope of maven we are discussing.
          If the patch presented here is against /dev-tools, and is to assist developers who like maven, then as
          I said before I am totally ok with this, but I'm only speaking for myself.

          Because maven is so many things, and due to Earwin's confusion, I think it would be good in general to
          add a README.txt to dev-tools anyway, that states what exactly it is (tools to assist lucene/solr developers,
          that aren't supported, its not bugs if they stop working, and will be deleted if they rot).

          Separately what you said about other code in trunk is totally true... for example its my opinion that there is
          a lot of code in lucene's contrib that should be moved out to something like apache-extras... currently lucene's
          contrib has to compile and pass tests or the build fails... there is definitely some stuff in there that is more
          "sandboxy", slows down lucene core development, but itself isnt getting much maintenance other than devs
          doing the minimum work to make them pass tests... and we should be keep other options in mind for stuff like this.

          Show
          Robert Muir added a comment - Chris: well thats the problem with maven, it tries to be too many things, a dependency management tool, a packaging system, a build system, ... So, thats why I said we have to just be very clear about which exact scope of maven we are discussing. If the patch presented here is against /dev-tools, and is to assist developers who like maven, then as I said before I am totally ok with this, but I'm only speaking for myself. Because maven is so many things, and due to Earwin's confusion, I think it would be good in general to add a README.txt to dev-tools anyway, that states what exactly it is (tools to assist lucene/solr developers, that aren't supported, its not bugs if they stop working, and will be deleted if they rot). Separately what you said about other code in trunk is totally true... for example its my opinion that there is a lot of code in lucene's contrib that should be moved out to something like apache-extras... currently lucene's contrib has to compile and pass tests or the build fails... there is definitely some stuff in there that is more "sandboxy", slows down lucene core development, but itself isnt getting much maintenance other than devs doing the minimum work to make them pass tests... and we should be keep other options in mind for stuff like this.
          Hide
          Earwin Burrfoot added a comment -

          Thanks, but I'm not the one confused here. : )

          Show
          Earwin Burrfoot added a comment - Thanks, but I'm not the one confused here. : )
          Hide
          Chris Male added a comment -

          Ant does many things too and we use it in a specific way so I see no problem defining what we intend our maven support to be for.

          So I'm feeling some consensus (fortunately I spoke too soon before) that if we target this toward being a development tool which is not
          forced upon any users / release managers.

          Is this okay with you Steven?

          A README.txt describing the scope of the dev-tools sounds appropriate irrespective of what happens here. I certainly wasn't aware of what
          their maintenance plan was.

          Show
          Chris Male added a comment - Ant does many things too and we use it in a specific way so I see no problem defining what we intend our maven support to be for. So I'm feeling some consensus (fortunately I spoke too soon before) that if we target this toward being a development tool which is not forced upon any users / release managers. Is this okay with you Steven? A README.txt describing the scope of the dev-tools sounds appropriate irrespective of what happens here. I certainly wasn't aware of what their maintenance plan was.
          Hide
          Ryan McKinley added a comment -

          Steve, great work with this patch – it takes care of all the previous concerns about our problematic maven support.

          With this patch, we now have:

          • testable maven artifacts
          • easy repo distribution
          • ant is still the build system

          The RM can choose to ignore the generate-maven-artifacts target and let someone else push the artifacts.

          As with most religious conflicts – I hope the resolution is not conversion, rather something that lets everyone to live (work) in peace.

          Show
          Ryan McKinley added a comment - Steve, great work with this patch – it takes care of all the previous concerns about our problematic maven support. With this patch, we now have: testable maven artifacts easy repo distribution ant is still the build system The RM can choose to ignore the generate-maven-artifacts target and let someone else push the artifacts. As with most religious conflicts – I hope the resolution is not conversion, rather something that lets everyone to live (work) in peace.
          Hide
          Chris A. Mattmann added a comment -

          +1 for Steve's patch, great work and you beat me to it.

          Show
          Chris A. Mattmann added a comment - +1 for Steve's patch, great work and you beat me to it.
          Hide
          Steve Rowe added a comment -

          Final version of trunk patch. Added CHANGES.txt entries; converted old solr-tika-parsers-<version> dependency to tika-parsers-0.8; and changed Solr snapshot artifact generation to use default uniqueVersion value (true), like Lucene's.

          Show
          Steve Rowe added a comment - Final version of trunk patch. Added CHANGES.txt entries; converted old solr-tika-parsers-<version> dependency to tika-parsers-0.8 ; and changed Solr snapshot artifact generation to use default uniqueVersion value ( true ), like Lucene's.
          Hide
          Steve Rowe added a comment -

          branch_3x patch. Maven build works, all tests pass, ant generate-maven-artifacts works.

          There is one extra private artifact required in order to build with Java 5 (trunk required Java 6, so doesn't have this issue): solr/contrib/clustering/lib/carrot2-core-3.4.2.jar. The artifact in Maven central repo was compiled with Java 6, but the private artifact was specially compiled with Java 5. I added populating the local repo with this .jar to the bootstrap profile, and pulled the POM from the artifact in Maven central, to enable transitive dependency resolution.

          Show
          Steve Rowe added a comment - branch_3x patch. Maven build works, all tests pass, ant generate-maven-artifacts works. There is one extra private artifact required in order to build with Java 5 (trunk required Java 6, so doesn't have this issue): solr/contrib/clustering/lib/carrot2-core-3.4.2.jar . The artifact in Maven central repo was compiled with Java 6, but the private artifact was specially compiled with Java 5. I added populating the local repo with this .jar to the bootstrap profile, and pulled the POM from the artifact in Maven central, to enable transitive dependency resolution.
          Hide
          Steve Rowe added a comment -

          I plan on committing these patches tomorrow.

          Show
          Steve Rowe added a comment - I plan on committing these patches tomorrow.
          Hide
          Robert Muir added a comment -

          as I said before... i really think the patches need to be against /dev-tools

          Show
          Robert Muir added a comment - as I said before... i really think the patches need to be against /dev-tools
          Hide
          Ryan McKinley added a comment -

          i really think the patches need to be against /dev-tools

          Sadly, given the way maven works, that is not possible. (without reinventing a crazy wheel)

          Previous attempts at fakey fakey pom generation have not been so good – this approach is better since it is testable and works well with core maven tools.

          Show
          Ryan McKinley added a comment - i really think the patches need to be against /dev-tools Sadly, given the way maven works, that is not possible. (without reinventing a crazy wheel) Previous attempts at fakey fakey pom generation have not been so good – this approach is better since it is testable and works well with core maven tools.
          Hide
          Robert Muir added a comment -

          Sorry, but introducing a parallel build system (somehow such that maven can "sneak" its way into being a 2nd supported build) is not acceptable to me.
          I feel this will slow down development.
          Putting it in /dev-tools clarifies the level of support: i don't want someone telling me "I broke the build" because these poms dont work if i refactor something.

          So I am -1 to the patch, unless it is in /dev-tools.
          if maven cannot work with its files in /dev-tools, perhaps a patch to maven itself is needed instead.

          Show
          Robert Muir added a comment - Sorry, but introducing a parallel build system (somehow such that maven can "sneak" its way into being a 2nd supported build) is not acceptable to me. I feel this will slow down development. Putting it in /dev-tools clarifies the level of support: i don't want someone telling me "I broke the build" because these poms dont work if i refactor something. So I am -1 to the patch, unless it is in /dev-tools. if maven cannot work with its files in /dev-tools, perhaps a patch to maven itself is needed instead.
          Hide
          David Smiley added a comment -

          Perhaps a compromise would be to put them in /dev-tools and have an ant task that will copy (or symbolically link them when possible) to where Maven wants them? Developers opposed to maven stuff can choose to never invoke this ant task and thus leave the maven stuff hidden.

          Show
          David Smiley added a comment - Perhaps a compromise would be to put them in /dev-tools and have an ant task that will copy (or symbolically link them when possible) to where Maven wants them? Developers opposed to maven stuff can choose to never invoke this ant task and thus leave the maven stuff hidden.
          Hide
          Robert Muir added a comment -

          David: sounds perfect to me, great idea.

          Show
          Robert Muir added a comment - David: sounds perfect to me, great idea.
          Hide
          Grant Ingersoll added a comment -

          Here's my suggestion: Let's put the stuff in and do a release. I am pretty sure that we have enough PMC members who want Maven that they can vote on said artifacts w/o those who don't want Maven to have to bother (even though they are the same artifacts). Also, Robert, I would specifically ask you, once Steve has done the work, assuming you are the RM, to honestly evaluate whether the production of the artifacts is a burden for you. If you don't want to, then I would suggest you do all the main RM work and you let one of us handle producing the artifacts to be released/voted on at the same time. If it really is a burden, then we can move it to dev-tools or external.

          And, for the record, there are more than two camps for Maven besides "die maven die" and "maven or die": I personally think distributing artifacts via the central repository and the explicit requirement to list out dependencies and their versions is a great service and significantly increases uptake of artifacts, but I can live just fine without the day to day crap that is the result of running "mvn XXXX" (where XXXX is not equal to one of the simpler things like compile or test). Let's not be confused about what we are doing when we say we are supporting Maven. We are just using it to publish artifacts. The only time that should ever need to be run is during a release (hence it doesn't slow down us deciding to do a release, it just slows down, marginally, the actual release process.)

          I totally agree we don't want two build systems and I am totally against switching to Maven as a build for all the reasons Robert is (parallel tests, etc.). All I have ever wanted is reliable published artifacts and I think we have enough support amongst committers to do that and we have strong enough interest in the community to warrant it.

          Show
          Grant Ingersoll added a comment - Here's my suggestion: Let's put the stuff in and do a release. I am pretty sure that we have enough PMC members who want Maven that they can vote on said artifacts w/o those who don't want Maven to have to bother (even though they are the same artifacts). Also, Robert, I would specifically ask you, once Steve has done the work, assuming you are the RM, to honestly evaluate whether the production of the artifacts is a burden for you. If you don't want to, then I would suggest you do all the main RM work and you let one of us handle producing the artifacts to be released/voted on at the same time. If it really is a burden, then we can move it to dev-tools or external. And, for the record, there are more than two camps for Maven besides "die maven die" and "maven or die": I personally think distributing artifacts via the central repository and the explicit requirement to list out dependencies and their versions is a great service and significantly increases uptake of artifacts, but I can live just fine without the day to day crap that is the result of running "mvn XXXX" (where XXXX is not equal to one of the simpler things like compile or test). Let's not be confused about what we are doing when we say we are supporting Maven. We are just using it to publish artifacts. The only time that should ever need to be run is during a release (hence it doesn't slow down us deciding to do a release, it just slows down, marginally, the actual release process.) I totally agree we don't want two build systems and I am totally against switching to Maven as a build for all the reasons Robert is (parallel tests, etc.). All I have ever wanted is reliable published artifacts and I think we have enough support amongst committers to do that and we have strong enough interest in the community to warrant it.
          Hide
          Robert Muir added a comment -

          If it really is a burden, then we can move it to dev-tools or external.

          Please re-read my comment above.

          I won't stand for two build systems to deal with during development. If all you care about is reliable published artifacts, then its not an issue
          if the stuff to generate that is in /dev-tools or mixed in with the code.

          I feel this is the maven users trying to sneak maven in as a fully supported build system, if not to then sneak it in as the only build system.

          I will do everything possible to stop this.

          Show
          Robert Muir added a comment - If it really is a burden, then we can move it to dev-tools or external. Please re-read my comment above. I won't stand for two build systems to deal with during development. If all you care about is reliable published artifacts, then its not an issue if the stuff to generate that is in /dev-tools or mixed in with the code. I feel this is the maven users trying to sneak maven in as a fully supported build system, if not to then sneak it in as the only build system. I will do everything possible to stop this.
          Hide
          Steve Rowe added a comment -

          I'm sensitive to the two-build-systems problem. While I like being able to produce local Maven artifacts, I agree that mixed build systems for new users will be confusing.

          I'm okay with putting the POMs themselves under dev-tools/. But I want the generate-maven-artifacts target to stay in Ant's build, not under dev-tools/, and a new target get-maven-poms added to the top-level build.xml; generate-maven-artifacts would depend on get-maven-poms. All pom.xml's would be added to svn:ignore entries where they are placed by get-maven-poms.

          Robert, would you agree to keeping generate-maven-artifacts and adding get-maven-poms?

          Show
          Steve Rowe added a comment - I'm sensitive to the two-build-systems problem. While I like being able to produce local Maven artifacts, I agree that mixed build systems for new users will be confusing. I'm okay with putting the POMs themselves under dev-tools/ . But I want the generate-maven-artifacts target to stay in Ant's build, not under dev-tools/ , and a new target get-maven-poms added to the top-level build.xml ; generate-maven-artifacts would depend on get-maven-poms . All pom.xml 's would be added to svn:ignore entries where they are placed by get-maven-poms . Robert, would you agree to keeping generate-maven-artifacts and adding get-maven-poms ?
          Hide
          Robert Muir added a comment -

          Steven, i totally dont mind if we change the whole way the ant build works with regards to maven, to depend on the dev-tools scenario.

          For example, you could have 'ant maven' that:
          1. copies the stuff from /dev-tools wherever it needs to be to make maven happy.
          2. does whatever is necessary (invoke maven, whatever) to make the artifacts.

          Or you could have 'ant maven' that only does part #1, but 'ant maven-artifacts' that does part #2, and depends on part #1.

          i just want it to be in /dev-tools, because if i see build.xml in a directory, i think its supported by Ant. If i see Makefile
          in a directory, I think its supported by Make, pom.xml by Maven, and so on.

          we could also add pom.xml to the svn:ignore for all the directories too, so that there are no problems with people
          using the maven framework in dev-tools.

          Show
          Robert Muir added a comment - Steven, i totally dont mind if we change the whole way the ant build works with regards to maven, to depend on the dev-tools scenario. For example, you could have 'ant maven' that: 1. copies the stuff from /dev-tools wherever it needs to be to make maven happy. 2. does whatever is necessary (invoke maven, whatever) to make the artifacts. Or you could have 'ant maven' that only does part #1, but 'ant maven-artifacts' that does part #2, and depends on part #1. i just want it to be in /dev-tools, because if i see build.xml in a directory, i think its supported by Ant. If i see Makefile in a directory, I think its supported by Make, pom.xml by Maven, and so on. we could also add pom.xml to the svn:ignore for all the directories too, so that there are no problems with people using the maven framework in dev-tools.
          Hide
          Steve Rowe added a comment - - edited

          A chat on IRC confirms that Robert is okay with my proposal:

          1. All POMs will be stored under dev-tools/maven/.
          2. generate-maven-artifacts will stay in the Ant build.
          3. A new get-maven-poms target will be added to the Ant build to copy the POMs from dev-tools/ to their target locations: in the top-level directory, and in each module's subdirectory under lucene/, solr/, and modules/ (trunk only).
          4. svn:ignore properties will be changed to include pom.xml in their target locations.
          5. generate-maven-artifacts will depend on get-maven-poms.

          I'll work up modified patches for trunk and branch_3x along these lines.

          Show
          Steve Rowe added a comment - - edited A chat on IRC confirms that Robert is okay with my proposal: All POMs will be stored under dev-tools/maven/ . generate-maven-artifacts will stay in the Ant build. A new get-maven-poms target will be added to the Ant build to copy the POMs from dev-tools/ to their target locations: in the top-level directory, and in each module's subdirectory under lucene/ , solr/ , and modules/ (trunk only). svn:ignore properties will be changed to include pom.xml in their target locations. generate-maven-artifacts will depend on get-maven-poms . I'll work up modified patches for trunk and branch_3x along these lines.
          Hide
          Steve Rowe added a comment -

          Patches implementing my proposal to place the Maven POMs in dev-tools/maven/ and add a new top-level Ant target get-maven-poms, which is invoked by generate-maven-artifacts. generate-maven-artifacts remains in the top-level build.xml, as well as in lucene/, solr/, and modules/ (trunk only).

          I couldn't figure out a way for generate-maven-artifacts under child directories lucene/, solr/ and modules/ to depend on the top-level get-maven-poms target, so I instead have generate-maven-artifacts in the child directories explicitly run the get-maven-poms target via the <ant> task. As a result, running generate-maven-artifacts from the top level will cause get-maven-poms to be run once for each child directory, but the repeated copy operation doesn't hurt anything, and the process is quick.

          Unless there are objections, I will commit this tomorrow.

          Show
          Steve Rowe added a comment - Patches implementing my proposal to place the Maven POMs in dev-tools/maven/ and add a new top-level Ant target get-maven-poms , which is invoked by generate-maven-artifacts . generate-maven-artifacts remains in the top-level build.xml , as well as in lucene/ , solr/ , and modules/ (trunk only). I couldn't figure out a way for generate-maven-artifacts under child directories lucene/ , solr/ and modules/ to depend on the top-level get-maven-poms target, so I instead have generate-maven-artifacts in the child directories explicitly run the get-maven-poms target via the <ant> task. As a result, running generate-maven-artifacts from the top level will cause get-maven-poms to be run once for each child directory, but the repeated copy operation doesn't hurt anything, and the process is quick. Unless there are objections, I will commit this tomorrow.
          Hide
          Robert Muir added a comment -

          +1, patch looks good.

          Show
          Robert Muir added a comment - +1, patch looks good.
          Hide
          Steve Rowe added a comment -

          Committed to trunk rev. 1061613, branch_3x rev. 1061612

          Show
          Steve Rowe added a comment - Committed to trunk rev. 1061613, branch_3x rev. 1061612
          Hide
          Steve Rowe added a comment -

          Uwe pointed out on IRC that the POMs under dev-tools/maven/ have a fixed version, and as it now is written, ant generate-maven-artifacts no longer respects the maven version property given on the command line.

          This should be fixed by converting the pom.xml files back into templates (and renaming the files accordingly), so that the version in the POMs can be modified during ant generate-maven-artifacts.

          ant get-maven-poms will be responsible for the substitution and renaming, in addition to the copying from dev-tools/maven/ that it currently does.

          Show
          Steve Rowe added a comment - Uwe pointed out on IRC that the POMs under dev-tools/maven/ have a fixed version, and as it now is written, ant generate-maven-artifacts no longer respects the maven version property given on the command line. This should be fixed by converting the pom.xml files back into templates (and renaming the files accordingly), so that the version in the POMs can be modified during ant generate-maven-artifacts . ant get-maven-poms will be responsible for the substitution and renaming, in addition to the copying from dev-tools/maven/ that it currently does.
          Hide
          Chris Male added a comment - - edited

          This should be fixed by converting the pom.xml files back into templates (and renaming the files accordingly), so that the version in the POMs can be modified during ant generate-maven-artifacts.

          Wasn't the fact we were using templates part of the problem that spawned this issue? I feel we've gone full circle here and are now back with maven templates that may or may not be maintained.

          Show
          Chris Male added a comment - - edited This should be fixed by converting the pom.xml files back into templates (and renaming the files accordingly), so that the version in the POMs can be modified during ant generate-maven-artifacts. Wasn't the fact we were using templates part of the problem that spawned this issue? I feel we've gone full circle here and are now back with maven templates that may or may not be maintained.
          Hide
          Steve Rowe added a comment -

          This should be fixed by converting the pom.xml files back into templates (and renaming the files accordingly), so that the version in the POMs can be modified during ant generate-maven-artifacts.

          Wasn't the fact we were using templates part of the problem that spawned this issue? I feel we've gone full circle here and are now back with maven templates that may or may not be maintained.

          Since the POMs don't represent an official build, the number of people using them to build will be small. It is this, IMHO, rather than whether the POMs are stored as templates under dev-tools/maven/, that will directly affect the quality/regularity of their maintenance.

          In fact, I think whether the POMs are stored as templates under dev-tools/maven/ is really a non-issue, since they won't be used there. The way it's set up right now, if you want to use the POMs to drive a build, is that you run ant get-maven-poms to copy them over to their target locations. The way it will be set up once I convert the POMs under dev-tools/maven/ to templates is that you run ant get-maven-poms to copy them over to their target locations, as well as naming them to pom.xml and interpolating the desired version. In other words, for the end-user, the experience will be exactly the same: you run ant get-maven-poms, and then you can use Maven to build/test/install artifacts.

          Templating the POMs will not change usability of them at all, so neither the number of people using them nor their effective maintenance will be affected by this change.

          All that said, I agree with you that maintenance is the big deal here, and I welcome other suggestions about how to simplify and enable that process.

          Show
          Steve Rowe added a comment - This should be fixed by converting the pom.xml files back into templates (and renaming the files accordingly), so that the version in the POMs can be modified during ant generate-maven-artifacts. Wasn't the fact we were using templates part of the problem that spawned this issue? I feel we've gone full circle here and are now back with maven templates that may or may not be maintained. Since the POMs don't represent an official build, the number of people using them to build will be small. It is this, IMHO, rather than whether the POMs are stored as templates under dev-tools/maven/ , that will directly affect the quality/regularity of their maintenance. In fact, I think whether the POMs are stored as templates under dev-tools/maven/ is really a non-issue, since they won't be used there. The way it's set up right now, if you want to use the POMs to drive a build, is that you run ant get-maven-poms to copy them over to their target locations. The way it will be set up once I convert the POMs under dev-tools/maven/ to templates is that you run ant get-maven-poms to copy them over to their target locations, as well as naming them to pom.xml and interpolating the desired version. In other words, for the end-user, the experience will be exactly the same: you run ant get-maven-poms , and then you can use Maven to build/test/install artifacts. Templating the POMs will not change usability of them at all, so neither the number of people using them nor their effective maintenance will be affected by this change. All that said, I agree with you that maintenance is the big deal here, and I welcome other suggestions about how to simplify and enable that process.
          Hide
          Steve Rowe added a comment -

          Rather than post a patch attachment for this change, I'll put the diffs here. I'll commit these shortly.

          First, all pom.xml files under dev-tools/maven/ will be renamed to pom.xml.template.

          Second, all pom.xml.template files will get this change (a representative version of it is given here - different original version for trunk's build.xml of course):

          -  <version>3.1-SNAPSHOT</version>
          +  <version>@version@</version>
          

          Third, the top-level build.xml gets this change (different version for trunk's build.xml of course):

          @@ -39,10 +39,15 @@
               </sequential>
             </target>
           
          +  <property name="version" value="3.1-SNAPSHOT"/>
             <target name="get-maven-poms" 
                     description="Copy Maven POMs from dev-tools/maven/ to their target locations">
               <copy todir=".">
                 <fileset dir="${basedir}/dev-tools/maven"/>
          +      <filterset begintoken="@" endtoken="@">
          +        <filter token="version" value="${version}"/>
          +      </filterset>
          +      <globmapper from="*.template" to="*"/>
               </copy>
             </target>
          
          Show
          Steve Rowe added a comment - Rather than post a patch attachment for this change, I'll put the diffs here. I'll commit these shortly. First, all pom.xml files under dev-tools/maven/ will be renamed to pom.xml.template . Second, all pom.xml.template files will get this change (a representative version of it is given here - different original version for trunk's build.xml of course): - <version>3.1-SNAPSHOT</version> + <version>@version@</version> Third, the top-level build.xml gets this change (different version for trunk's build.xml of course): @@ -39,10 +39,15 @@ </sequential> </target> + <property name="version" value="3.1-SNAPSHOT"/> <target name="get-maven-poms" description="Copy Maven POMs from dev-tools/maven/ to their target locations"> <copy todir="."> <fileset dir="${basedir}/dev-tools/maven"/> + <filterset begintoken="@" endtoken="@"> + <filter token="version" value="${version}"/> + </filterset> + <globmapper from="*.template" to="*"/> </copy> </target>
          Hide
          Steve Rowe added a comment -

          Committed the template interpolation and renaming patches from my previous comment here, to trunk rev. 1062111 and branch_3x rev. r1062112

          Show
          Steve Rowe added a comment - Committed the template interpolation and renaming patches from my previous comment here, to trunk rev. 1062111 and branch_3x rev. r1062112
          Hide
          David Smiley added a comment -

          Since the POMs don't represent an official build, the number of people using them to build will be small. It is this, IMHO, rather than whether the POMs are stored as templates under dev-tools/maven/, that will directly affect the quality/regularity of their maintenance.

          FWIW, I'll start using the generated POMs for my builds (aside from my use of IntelliJ for day to day development). If there are ever problems then I'll of course report/fix them. Thanks so much Steven for creating this alternative build system (one that I prefer).

          Show
          David Smiley added a comment - Since the POMs don't represent an official build, the number of people using them to build will be small. It is this, IMHO, rather than whether the POMs are stored as templates under dev-tools/maven/, that will directly affect the quality/regularity of their maintenance. FWIW, I'll start using the generated POMs for my builds (aside from my use of IntelliJ for day to day development). If there are ever problems then I'll of course report/fix them. Thanks so much Steven for creating this alternative build system (one that I prefer).
          Hide
          Steve Rowe added a comment -

          Since the POMs don't represent an official build, the number of people using them to build will be small. It is this, IMHO, rather than whether the POMs are stored as templates under dev-tools/maven/, that will directly affect the quality/regularity of their maintenance.

          FWIW, I'll start using the generated POMs for my builds (aside from my use of IntelliJ for day to day development). If there are ever problems then I'll of course report/fix them. Thanks so much Steven for creating this alternative build system (one that I prefer).

          Cool! The more the merrier.

          Show
          Steve Rowe added a comment - Since the POMs don't represent an official build, the number of people using them to build will be small. It is this, IMHO, rather than whether the POMs are stored as templates under dev-tools/maven/, that will directly affect the quality/regularity of their maintenance. FWIW, I'll start using the generated POMs for my builds (aside from my use of IntelliJ for day to day development). If there are ever problems then I'll of course report/fix them. Thanks so much Steven for creating this alternative build system (one that I prefer). Cool! The more the merrier.
          Hide
          Grant Ingersoll added a comment -

          Bulk close for 3.1

          Show
          Grant Ingersoll added a comment - Bulk close for 3.1

            People

            • Assignee:
              Steve Rowe
              Reporter:
              Steve Rowe
            • Votes:
              6 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development