Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 3.1, 4.0-ALPHA
    • Fix Version/s: 3.1, 4.0-ALPHA
    • Component/s: general/build
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      Setting up Lucene/Solr in IntelliJ IDEA or Eclipse can be time-consuming.

      The attached patches add a new top level directory dev-tools/ with sub-dirs idea/ and eclipse/ containing basic setup files for trunk, as well as top-level ant targets named "idea" and "eclipse" that copy these files into the proper locations. This arrangement avoids the messiness attendant to in-place project configuration files directly checked into source control.

      The IDEA configuration includes modules for Lucene and Solr, each Lucene and Solr contrib, and each analysis module. A JUnit run configuration per module is included.

      The Eclipse configuration includes a source entry for each source/test/resource location and classpath setup: a library entry for each jar.

      For IDEA, once ant idea has been run, the only configuration that must be performed manually is configuring the project-level JDK. For Eclipse, once ant eclipse has been run, the user has to refresh the project (right-click on the project and choose Refresh).

      If these patches is committed, Subversion svn:ignore properties should be added/modified to ignore the destination IDEA and Eclipse configuration locations.

      Iam Jambour has written up on the Lucene wiki a detailed set of instructions for applying the 3.X branch patch for IDEA: http://wiki.apache.org/lucene-java/HowtoConfigureIntelliJ

      1. LUCENE-2611.patch
        66 kB
        Steve Rowe
      2. LUCENE-2611.patch
        69 kB
        Steve Rowe
      3. LUCENE-2611-branch-3x.patch
        68 kB
        Steve Rowe
      4. LUCENE-2611.patch
        72 kB
        Steve Rowe
      5. LUCENE-2611_test.patch
        3 kB
        Robert Muir
      6. LUCENE-2611_test.patch
        7 kB
        Robert Muir
      7. LUCENE-2611_test.patch
        12 kB
        Steve Rowe
      8. LUCENE-2611.patch
        71 kB
        Steve Rowe
      9. LUCENE-2611-branch-3x.patch
        69 kB
        Steve Rowe
      10. LUCENE-2611_test.patch
        16 kB
        Robert Muir
      11. LUCENE-2611_test_2.patch
        2 kB
        Steve Rowe
      12. LUCENE-2611.patch
        75 kB
        Steve Rowe
      13. LUCENE-2611-branch-3x.patch
        71 kB
        Steve Rowe
      14. LUCENE-2611_mkdir.patch
        0.6 kB
        Robert Muir
      15. LUCENE-2611.patch
        74 kB
        Steve Rowe
      16. LUCENE-2611-branch-3x.patch
        73 kB
        Steve Rowe
      17. LUCENE-2611.patch
        74 kB
        Steve Rowe
      18. LUCENE-2611_eclipse.patch
        13 kB
        Robert Muir
      19. LUCENE-2611.patch
        83 kB
        Steve Rowe
      20. LUCENE-2611-branch-3x.patch
        82 kB
        Steve Rowe
      21. LUCENE-2611-part2.patch
        28 kB
        Steve Rowe
      22. LUCENE-2611-branch-3x-part2.patch
        27 kB
        Steve Rowe
      23. utf8.patch
        1 kB
        David Smiley
      24. LUCENE-2611_intellij_fix_codestyle.patch
        9 kB
        David Smiley
      25. LUCENE-2611_intellij_fix_codestyle.patch
        8 kB
        Steve Rowe

        Activity

        Hide
        Grant Ingersoll added a comment -

        Bulk close for 3.1

        Show
        Grant Ingersoll added a comment - Bulk close for 3.1
        Hide
        Steve Rowe added a comment -

        Committed code style fixes to branch_3x and trunk. Thanks David!

        Show
        Steve Rowe added a comment - Committed code style fixes to branch_3x and trunk. Thanks David!
        Hide
        Steve Rowe added a comment -

        David,

        I made a couple of changes:

        1. I got rid of the global code style file. I don't see the point in maintaining two copies of the same file (except for the name). People who want a global code style can likely manage conversion of the per-project code style file.
        2. I simplified the idea target in the top-level build.xml and got rid of vestigial environment and platform variables that were there to support per-platform global style file installation instructions.

        Committing shortly.

        Show
        Steve Rowe added a comment - David, I made a couple of changes: I got rid of the global code style file. I don't see the point in maintaining two copies of the same file (except for the name). People who want a global code style can likely manage conversion of the per-project code style file. I simplified the idea target in the top-level build.xml and got rid of vestigial environment and platform variables that were there to support per-platform global style file installation instructions. Committing shortly.
        Hide
        David Smiley added a comment -

        This patch uses a project-specified code style. This simplified the build.xml a little since we don't need to tell people to copy their intellij codestyle file to a path OS-specific. I also looked carefully at the code style to ensure it was as canonical as it should be (e.g. "use same indents = true, continuation indent size = 4 even for groovy,java,xml), and don't need the parent scheme name. That was a little time-consuming as I restarted IntelliJ many times to see how it reacted to little changes. I updated the stand-alone codestyle accordingly too... someone might want to install a global named codestyle but it's not required.

        Show
        David Smiley added a comment - This patch uses a project-specified code style. This simplified the build.xml a little since we don't need to tell people to copy their intellij codestyle file to a path OS-specific. I also looked carefully at the code style to ensure it was as canonical as it should be (e.g. "use same indents = true, continuation indent size = 4 even for groovy,java,xml), and don't need the parent scheme name. That was a little time-consuming as I restarted IntelliJ many times to see how it reacted to little changes. I updated the stand-alone codestyle accordingly too... someone might want to install a global named codestyle but it's not required.
        Hide
        David Smiley added a comment -

        Yes.

        Show
        David Smiley added a comment - Yes.
        Hide
        Steve Rowe added a comment -

        David, I like the idea. Are project-level code styles stored in the .idea/ directory along with other configuration?

        Show
        Steve Rowe added a comment - David, I like the idea. Are project-level code styles stored in the .idea/ directory along with other configuration?
        Hide
        David Smiley added a comment -

        Steven, I think the IntelliJ IDEA configuration should use a project code-style. One of the things that annoys me about IntelliJ's code style configuration screen is that it's confusing--it kind of suggests that you can configure your project to point to an IDE-wide named code style. But you can't. There's a global style in effect, and you can switch among them by name, and there's your project's code style which is optional. I don't want to have to bring up this dialog, switch to the lucene code-style, then click the "copy to project" button. This will stream-line the setup too, since you don't need to create an global named code style, which isn't very useful.

        Show
        David Smiley added a comment - Steven, I think the IntelliJ IDEA configuration should use a project code-style. One of the things that annoys me about IntelliJ's code style configuration screen is that it's confusing--it kind of suggests that you can configure your project to point to an IDE-wide named code style. But you can't. There's a global style in effect, and you can switch among them by name, and there's your project's code style which is optional. I don't want to have to bring up this dialog, switch to the lucene code-style, then click the "copy to project" button. This will stream-line the setup too, since you don't need to create an global named code style, which isn't very useful.
        Hide
        Steve Rowe added a comment -

        I couldn't compile most of the non-english languages when I updated my 3x branch recently.

        David,

        IntelliJ has a project-wide encoding setting: Settings - Project Settings | File Encoding (see http://jetbrains.dzone.com/articles/new-approach-encoding for details). Mine is set to UTF-8. Is yours set to something different? I suspect that IntelliJ automatically uses this setting when invoking javac, and that's why I never had this problem.

        Show
        Steve Rowe added a comment - I couldn't compile most of the non-english languages when I updated my 3x branch recently. David, IntelliJ has a project-wide encoding setting: Settings - Project Settings | File Encoding (see http://jetbrains.dzone.com/articles/new-approach-encoding for details). Mine is set to UTF-8. Is yours set to something different? I suspect that IntelliJ automatically uses this setting when invoking javac , and that's why I never had this problem.
        Hide
        Steve Rowe added a comment - - edited

        edit: escaped the * metacharacter so that it appears as itself rather than switching to bold format.

        For some reason it was missed that javac should be invoked with "-encoding utf-8" options.

        Thanks for bringing this up - I hadn't realized that javac defaulted to the platform encoding. I've committed this part of your patch to both trunk and branch_3x.

        I also included a build.xml tweak that overwrites the .iml files

        I didn't commit this part of your patch, because I don't think it's a good idea to only overwrite the .iml files, and not also the .idea/.xml files, since the two have to be in sync, e.g. when there are structural changes.

        Right now there is a clean-idea task that can be used to serve this function. The .idea directory is where IntelliJ stores stuff like shelved changes, and it would be really bad to automatically delete that stuff as part of updating IntelliJ configuration, so that directory is never automatically overwritten.

        One technical note about this part of your patch:

        <copy todir=".">
          <fileset dir="dev-tools/idea">
            <exclude name="Intellij-Lucene-Codestyle.xml"/>
        +   <exclude name="*.iml"/>
          </fileset>
          </copy>
        + <copy todir="." overwrite="true">
        +   <fileset dir="dev-tools/idea">
        +     <include name="*.iml"/>
        +   </fileset>
        + </copy>
        

        *.iml does not refer to all .iml files in all sub-directories, but rather only those in the top-level directory. You want **/*.iml to catch all of them recursively.

        Show
        Steve Rowe added a comment - - edited edit : escaped the * metacharacter so that it appears as itself rather than switching to bold format. For some reason it was missed that javac should be invoked with "-encoding utf-8" options. Thanks for bringing this up - I hadn't realized that javac defaulted to the platform encoding. I've committed this part of your patch to both trunk and branch_3x. I also included a build.xml tweak that overwrites the .iml files I didn't commit this part of your patch, because I don't think it's a good idea to only overwrite the .iml files, and not also the .idea/.xml files, since the two have to be in sync, e.g. when there are structural changes. Right now there is a clean-idea task that can be used to serve this function. The .idea directory is where IntelliJ stores stuff like shelved changes, and it would be really bad to automatically delete that stuff as part of updating IntelliJ configuration, so that directory is never automatically overwritten. One technical note about this part of your patch: <copy todir= "." > <fileset dir= "dev-tools/idea" > <exclude name= "Intellij-Lucene-Codestyle.xml" /> + <exclude name= "*.iml" /> </fileset> </copy> + <copy todir= "." overwrite= "true" > + <fileset dir= "dev-tools/idea" > + <include name= "*.iml" /> + </fileset> + </copy> *.iml does not refer to all .iml files in all sub-directories, but rather only those in the top-level directory. You want **/*.iml to catch all of them recursively.
        Hide
        David Smiley added a comment -

        For some reason it was missed that javac should be invoked with "-encoding utf-8" options. I couldn't compile most of the non-english languages when I updated my 3x branch recently. Attached is a patch adding this. I also included a build.xml tweak that overwrites the .iml files... feel free to leave that out or use it if you like it.

        Show
        David Smiley added a comment - For some reason it was missed that javac should be invoked with "-encoding utf-8" options. I couldn't compile most of the non-english languages when I updated my 3x branch recently. Attached is a patch adding this. I also included a build.xml tweak that overwrites the .iml files... feel free to leave that out or use it if you like it.
        Hide
        Steve Rowe added a comment -

        And perhaps the copyright setup should be set up for ASL.

        I've used the copyright plugin a lot and its a great way to ensure that the ASL is added to any new files. Might be useful to add it to reduce the hassle for new contributors.

        Committed IntelliJ IDEA Copyright Plugin configuration for the Apache Software Licence: trunk rev. 1059199, branch_3x rev. 1059200

        Show
        Steve Rowe added a comment - And perhaps the copyright setup should be set up for ASL. I've used the copyright plugin a lot and its a great way to ensure that the ASL is added to any new files. Might be useful to add it to reduce the hassle for new contributors. Committed IntelliJ IDEA Copyright Plugin configuration for the Apache Software Licence: trunk rev. 1059199, branch_3x rev. 1059200
        Hide
        Steve Rowe added a comment -

        I don't know why the vcs.xml change isn't working for you, but it's absolutely wonderful for the commit log history to show the JIRA references as links that work.

        I agree, that would be nice.

        FWIW, I'm using IntelliJ 10.

        I'm running both ATM, in part to insure that the configuration provided by this issue works under both IntelliJ 9 and 10. I haven't tried the log comment issue auto-linkification yet under IntelliJ 10.

        I do see auto-linkified issue IDs in the Repository Changes view, as well as in the [Version Control | Subversion | Show History] view in IntelliJ 9 - very nice! (Just not in the log comment editor or in the svnbar plugin's "SVN Diff" popup.)

        Show
        Steve Rowe added a comment - I don't know why the vcs.xml change isn't working for you, but it's absolutely wonderful for the commit log history to show the JIRA references as links that work. I agree, that would be nice. FWIW, I'm using IntelliJ 10. I'm running both ATM, in part to insure that the configuration provided by this issue works under both IntelliJ 9 and 10. I haven't tried the log comment issue auto-linkification yet under IntelliJ 10. I do see auto-linkified issue IDs in the Repository Changes view, as well as in the [Version Control | Subversion | Show History] view in IntelliJ 9 - very nice! (Just not in the log comment editor or in the svnbar plugin's "SVN Diff" popup.)
        Hide
        Steve Rowe added a comment -

        I've used the copyright plugin a lot and its a great way to ensure that the ASL is added to any new files. Might be useful to add it to reduce the hassle for new contributors.

        OK, I'll investigate.

        Show
        Steve Rowe added a comment - I've used the copyright plugin a lot and its a great way to ensure that the ASL is added to any new files. Might be useful to add it to reduce the hassle for new contributors. OK, I'll investigate.
        Hide
        David Smiley added a comment -

        I don't know why the vcs.xml change isn't working for you, but it's absolutely wonderful for the commit log history to show the JIRA references as links that work. FWIW, I'm using IntelliJ 10.

        I understand RE workspace.xml.

        Show
        David Smiley added a comment - I don't know why the vcs.xml change isn't working for you, but it's absolutely wonderful for the commit log history to show the JIRA references as links that work. FWIW, I'm using IntelliJ 10. I understand RE workspace.xml.
        Hide
        Chris Male added a comment - - edited

        I'm not sure it's a good idea to add copyright setup for ASL - I don't know enough about what this plugin does.

        I've used the copyright plugin a lot and its a great way to ensure that the ASL is added to any new files. Might be useful to add it to reduce the hassle for new contributors.

        Show
        Chris Male added a comment - - edited I'm not sure it's a good idea to add copyright setup for ASL - I don't know enough about what this plugin does. I've used the copyright plugin a lot and its a great way to ensure that the ASL is added to any new files. Might be useful to add it to reduce the hassle for new contributors.
        Hide
        Steve Rowe added a comment -

        Hi David,

        Thanks for the input.

        I don't think another issue is necessary.

        I added the .idea/vcs.xml change to auto-linkify issues in log comments. I didn't know this option existed. Where does it do the auto-linkification? I don't see it in the log comment editor, and I also don't see it when I use browse an individual file's log messages (using the popup from the svnbar plugin toolbar icon).

        But I did not add the .idea/workspace.xml change you propose (ignoring .idea/ and .iml files), because those files are already ignored via svn:ignore properties. When I added them, nothing changed for me - the files still show up in the project tree view greyed out, just as they did before I added the option.

        I'm not sure it's a good idea to add copyright setup for ASL - I don't know enough about what this plugin does.

        Show
        Steve Rowe added a comment - Hi David, Thanks for the input. I don't think another issue is necessary. I added the .idea/vcs.xml change to auto-linkify issues in log comments. I didn't know this option existed. Where does it do the auto-linkification? I don't see it in the log comment editor, and I also don't see it when I use browse an individual file's log messages (using the popup from the svnbar plugin toolbar icon). But I did not add the .idea/workspace.xml change you propose (ignoring .idea/ and .iml files), because those files are already ignored via svn:ignore properties. When I added them, nothing changed for me - the files still show up in the project tree view greyed out, just as they did before I added the option. I'm not sure it's a good idea to add copyright setup for ASL - I don't know enough about what this plugin does.
        Hide
        David Smiley added a comment -

        Steven,
        I don't know if another issue should be created but there's some extra additions to the IntelliJ setup that would be nice. in vcs.xml, add this:

        <component name="IssueNavigationConfiguration">
            <option name="links">
              <list>
                <IssueNavigationLink>
                  <option name="issueRegexp" value="[A-Z]+\-\d+" />
                  <option name="linkRegexp" value="http://issues.apache.org/jira/browse/$0" />
                </IssueNavigationLink>
              </list>
            </option>
          </component>
        

        And in workspace.xml, /project/component[@name="ChangeListManager"]/ add

            <ignored path=".idea/" />
            <ignored mask="*.iml" />
        

        And perhaps the copyright setup should be set up for ASL.

        Show
        David Smiley added a comment - Steven, I don't know if another issue should be created but there's some extra additions to the IntelliJ setup that would be nice. in vcs.xml, add this: <component name= "IssueNavigationConfiguration" > <option name= "links" > <list> <IssueNavigationLink> <option name= "issueRegexp" value= "[A-Z]+\-\d+" /> <option name= "linkRegexp" value= "http://issues.apache.org/jira/browse/$0" /> </IssueNavigationLink> </list> </option> </component> And in workspace.xml, /project/component [@name="ChangeListManager"] / add <ignored path= ".idea/" /> <ignored mask= "*.iml" /> And perhaps the copyright setup should be set up for ASL.
        Hide
        Steve Rowe added a comment -

        Patch for branch_3x, also without the same Subversion moves. Committing shortly.

        Show
        Steve Rowe added a comment - Patch for branch_3x, also without the same Subversion moves. Committing shortly.
        Hide
        Steve Rowe added a comment -

        After Robert's changes from SOLR-2299 to allow Eclipse to run all Lucene/Solr tests, the IntelliJ config in dev-tools/ is out of sync. This patch fixes that by moving solr/src/test/test-files/ to solr/src/test-files/, so that IntelliJ can specify solr/src/test-files/ as a content root (IntelliJ doesn't allow nested content roots, and solr/src/test/ is already a content root) – otherwise, test-files/ is copied over to the build directory, rather than just its contents.

        solr/contrib/analysis-extras/src/test/test-files/ is also moved up one level in the same fashion.

        This patch doesn't contain the Subversion move operations - first run svn mv solr/src/test/test-files solr/src/ ; svn mv solr/contrib/analysis-extras/src/test/test-files solr/contrib/analysis-extras/src/, then apply the patch.

        The patch also fixes up the Eclipse configuration in dev-tools/eclipse/.

        The patch also sets the Java language level on all Solr modules to 6.0. (The default language level is 5.0.)

        Committing shortly.

        Show
        Steve Rowe added a comment - After Robert's changes from SOLR-2299 to allow Eclipse to run all Lucene/Solr tests, the IntelliJ config in dev-tools/ is out of sync. This patch fixes that by moving solr/src/test/test-files/ to solr/src/test-files/ , so that IntelliJ can specify solr/src/test-files/ as a content root (IntelliJ doesn't allow nested content roots, and solr/src/test/ is already a content root) – otherwise, test-files/ is copied over to the build directory, rather than just its contents. solr/contrib/analysis-extras/src/test/test-files/ is also moved up one level in the same fashion. This patch doesn't contain the Subversion move operations - first run svn mv solr/src/test/test-files solr/src/ ; svn mv solr/contrib/analysis-extras/src/test/test-files solr/contrib/analysis-extras/src/ , then apply the patch. The patch also fixes up the Eclipse configuration in dev-tools/eclipse/ . The patch also sets the Java language level on all Solr modules to 6.0. (The default language level is 5.0.) Committing shortly.
        Hide
        David Smiley added a comment -

        Lance,
        The problem with the git repo at apache is that it doesn't properly maintain Lucene & Solr's history prior to the merger. I'd love to use git but this is a show-stopper. I asked the apache maintainers of the repo and they are aware of this issue but they didn't have the bandwidth to try to fix it. It'd probably be pretty hard.

        Show
        David Smiley added a comment - Lance, The problem with the git repo at apache is that it doesn't properly maintain Lucene & Solr's history prior to the merger. I'd love to use git but this is a show-stopper. I asked the apache maintainers of the repo and they are aware of this issue but they didn't have the bandwidth to try to fix it. It'd probably be pretty hard.
        Hide
        Lance Norskog added a comment -

        About making it easier to contribute with patches: what makes this easy is Git or Mercurial, not an IDE. If you make a local Git repository matching your checkout you can make separate branches for each project. Git is way too flexible and confusing, but I can't live without it now.

        You can do a full GIT checkout from the links on git.apache.org. There is a nice GIT extension for Eclipse that makes it really easy to manage branches.

        Cheers, Lance.

        Show
        Lance Norskog added a comment - About making it easier to contribute with patches: what makes this easy is Git or Mercurial, not an IDE. If you make a local Git repository matching your checkout you can make separate branches for each project. Git is way too flexible and confusing, but I can't live without it now. You can do a full GIT checkout from the links on git.apache.org. There is a nice GIT extension for Eclipse that makes it really easy to manage branches. Cheers, Lance.
        Hide
        Lance Norskog added a comment -

        Robert, thank you for this grand effort. After a few times, I can set it up faster now but it's still a pain.

        Now, the last thing you want to hear I use Eclipse text search a lot and it's much faster when a code based is split between a few projects instead of one large project. Maybe I can help on that one.

        Show
        Lance Norskog added a comment - Robert, thank you for this grand effort. After a few times, I can set it up faster now but it's still a pain. Now, the last thing you want to hear I use Eclipse text search a lot and it's much faster when a code based is split between a few projects instead of one large project. Maybe I can help on that one.
        Hide
        Robert Muir added a comment -

        ok, you should be able to checkout a project with subclipse (svn location: https://svn.apache.org/repos/asf/lucene/dev/trunk)
        After the svn checkout is done, run 'ant eclipse'. Then refresh your project... and things should work.

        I'll backport to 3.x and try to improve the docs etc later.

        Show
        Robert Muir added a comment - ok, you should be able to checkout a project with subclipse (svn location: https://svn.apache.org/repos/asf/lucene/dev/trunk ) After the svn checkout is done, run 'ant eclipse'. Then refresh your project... and things should work. I'll backport to 3.x and try to improve the docs etc later.
        Hide
        Robert Muir added a comment -

        I have the core tests working... the main reason it is taking so long, is that I am fixing our ant build to be like lucene's
        and copy test resources to the classpath.. this way it won't break once I am finished.

        I have the solr contrib tests working in another checkout but its hackish... just trying to get that part reasonable too.
        Hope to get this resolved today!

        Show
        Robert Muir added a comment - I have the core tests working... the main reason it is taking so long, is that I am fixing our ant build to be like lucene's and copy test resources to the classpath.. this way it won't break once I am finished. I have the solr contrib tests working in another checkout but its hackish... just trying to get that part reasonable too. Hope to get this resolved today!
        Hide
        Steve Rowe added a comment -

        Hi Adeel,

        will there be new instructions in HowToContrib section for eclipse and idea

        Yes, I've added a link from both Solr's and Lucene's HowToContribute to HowtoConfigureIntelliJ

        Robert Muir is working on getting all Lucene/Solr tests to run under Eclipse. Once he's got them running, I believe he plans on committing the Eclipse patch on this issue. The Eclipse instructions on the wiki should be cleaned up at that point too.

        i am not able to run individual tests and i have tried all the possible ways pointed out in solr wiki (e.g. ant -DtestCase ....)

        That should be ant -Dtestcase .... - note that the "c" in "testcase" is lowercase. Have you seen RunningTests?

        just thought ill make the voice of people, who want to contribute but just cant get past the initial setting things up obstacle, heard.

        Thanks for your feedback!

        Show
        Steve Rowe added a comment - Hi Adeel, will there be new instructions in HowToContrib section for eclipse and idea Yes, I've added a link from both Solr's and Lucene's HowToContribute to HowtoConfigureIntelliJ Robert Muir is working on getting all Lucene/Solr tests to run under Eclipse. Once he's got them running, I believe he plans on committing the Eclipse patch on this issue. The Eclipse instructions on the wiki should be cleaned up at that point too. i am not able to run individual tests and i have tried all the possible ways pointed out in solr wiki (e.g. ant -DtestCase ....) That should be ant -Dtestcase .... - note that the "c" in "testcase" is lowercase. Have you seen RunningTests ? just thought ill make the voice of people, who want to contribute but just cant get past the initial setting things up obstacle, heard. Thanks for your feedback!
        Hide
        Adeel Qureshi added a comment -

        i recently tried setting up solr dev environment in my eclipse and i am really glad to see Robert Muir's comments regarding how difficult it is to get it all setup and working in these ide's because it took me about 3-4 redo everything attempts and about a week to get it all setup simply to be able to run ant test and even then some tests still failed (very few). i also had all kinds of problems in running specific tests instead of every single test. so anyways with all these new updates you guys have made .. did all that made anything easier. will there be new instructions in HowToContrib section for eclipse and idea .. also another thing to mention .. i setup mine on eclipse 3.5 but from HowToContrib i had to do steps mentioned for eclipse 3.6 as well .. also as i mentioned before i am not able to run individual tests and i have tried all the possible ways pointed out in solr wiki (e.g. ant -DtestCase ....) ..

        just thought ill make the voice of people, who want to contribute but just cant get past the initial setting things up obstacle, heard.

        Thanks
        Adeel

        Show
        Adeel Qureshi added a comment - i recently tried setting up solr dev environment in my eclipse and i am really glad to see Robert Muir's comments regarding how difficult it is to get it all setup and working in these ide's because it took me about 3-4 redo everything attempts and about a week to get it all setup simply to be able to run ant test and even then some tests still failed (very few). i also had all kinds of problems in running specific tests instead of every single test. so anyways with all these new updates you guys have made .. did all that made anything easier. will there be new instructions in HowToContrib section for eclipse and idea .. also another thing to mention .. i setup mine on eclipse 3.5 but from HowToContrib i had to do steps mentioned for eclipse 3.6 as well .. also as i mentioned before i am not able to run individual tests and i have tried all the possible ways pointed out in solr wiki (e.g. ant -DtestCase ....) .. just thought ill make the voice of people, who want to contribute but just cant get past the initial setting things up obstacle, heard. Thanks Adeel
        Hide
        Steve Rowe added a comment -

        Hi Erick,

        I just tried this on my Windows box and it works perfectly, even to running tests. The only thing I had to do was open the project from IntelliJ and set the compiler, just as expected. Well, and wait for IntelliJ to finish indexing things <G>...

        Sweet!

        I really think this will make it MUCH easier for people to contribute, I always dread setting up any IDE for a new project since it usually takes seemingly forever. Nice work!

        Thanks. The key, of course, is keeping it in synch with the project structure. I plan on writing up maintenance instructions the next time this is required, so that others can more easily contribute to the effort.

        I'll try it on my Mac just for yucks...

        Cool, let me know if you notice anything amiss.

        Show
        Steve Rowe added a comment - Hi Erick, I just tried this on my Windows box and it works perfectly, even to running tests. The only thing I had to do was open the project from IntelliJ and set the compiler, just as expected. Well, and wait for IntelliJ to finish indexing things <G>... Sweet! I really think this will make it MUCH easier for people to contribute, I always dread setting up any IDE for a new project since it usually takes seemingly forever. Nice work! Thanks. The key, of course, is keeping it in synch with the project structure. I plan on writing up maintenance instructions the next time this is required, so that others can more easily contribute to the effort. I'll try it on my Mac just for yucks... Cool, let me know if you notice anything amiss.
        Hide
        Erick Erickson added a comment -

        Steven:

        I just tried this on my Windows box and it works perfectly, even to running tests. The only thing I had to do was open the project from IntelliJ and set the compiler, just as expected. Well, and wait for IntelliJ to finish indexing things <G>...

        Sweet!

        I really think this will make it MUCH easier for people to contribute, I always dread setting up any IDE for a new project since it usually takes seemingly forever. Nice work! I'll try it on my Mac just for yucks...

        Erick

        Show
        Erick Erickson added a comment - Steven: I just tried this on my Windows box and it works perfectly, even to running tests. The only thing I had to do was open the project from IntelliJ and set the compiler, just as expected. Well, and wait for IntelliJ to finish indexing things <G>... Sweet! I really think this will make it MUCH easier for people to contribute, I always dread setting up any IDE for a new project since it usually takes seemingly forever. Nice work! I'll try it on my Mac just for yucks... Erick
        Hide
        Steve Rowe added a comment -

        branch_3x version of IntelliJ config files, including codestyle addition. Committing shortly.

        Show
        Steve Rowe added a comment - branch_3x version of IntelliJ config files, including codestyle addition. Committing shortly.
        Hide
        Steve Rowe added a comment -

        Added IntelliJ codestyle definition and instructions for putting it in the correct location. Committing shortly.

        Show
        Steve Rowe added a comment - Added IntelliJ codestyle definition and instructions for putting it in the correct location. Committing shortly.
        Hide
        Robert Muir added a comment -

        I just realized my comment probably makes no sense if you don't use eclipse...
        but eclipse's classpath works just likes java's... you can't add directories of jar files really.
        I think this is just a fundamental difference from IDEA?

        so if someone changes a jar file, it breaks the project's build until you reconfigure the classpath anyway.
        i'm thinking the easiest is to just keep it as a pure copy, so someone will fix their eclipse build
        on any structural build change (jar or not), copy their .classpath to dot.classpath, and commit it.

        anyway, how far away is your IDEA patch? if you are maintaining it on the issue, at some point
        maybe it would make sense to just commit it?

        I'd really like to see our howtocontribute just have simple one-liners for these IDEs rather than complicated
        instructions... e.g. svn checkout and run 'ant idea' or 'ant eclipse'

        Show
        Robert Muir added a comment - I just realized my comment probably makes no sense if you don't use eclipse... but eclipse's classpath works just likes java's... you can't add directories of jar files really. I think this is just a fundamental difference from IDEA? so if someone changes a jar file, it breaks the project's build until you reconfigure the classpath anyway. i'm thinking the easiest is to just keep it as a pure copy, so someone will fix their eclipse build on any structural build change (jar or not), copy their .classpath to dot.classpath, and commit it. anyway, how far away is your IDEA patch? if you are maintaining it on the issue, at some point maybe it would make sense to just commit it? I'd really like to see our howtocontribute just have simple one-liners for these IDEs rather than complicated instructions... e.g. svn checkout and run 'ant idea' or 'ant eclipse'
        Hide
        Robert Muir added a comment -

        Any reason why scripting wouldn't work?

        Scripting actually makes maintenance worse in this case, because currently to update it, someone can just copy
        their .classpath to it.

        If we script it, then if we need to update it (e.g. move a contrib to a module), then this becomes a fragile
        manual editing process.

        Show
        Robert Muir added a comment - Any reason why scripting wouldn't work? Scripting actually makes maintenance worse in this case, because currently to update it, someone can just copy their .classpath to it. If we script it, then if we need to update it (e.g. move a contrib to a module), then this becomes a fragile manual editing process.
        Hide
        Steve Rowe added a comment -

        here's the eclipse part (following the same conventions of your patch).

        basically for eclipse, getting things to work is just setting up the classpath and setting the whole project to UTF-8... but this takes a while even if you know everything you need to do.

        The IDEA configuration sets up dependencies as whole jar directories, but the setup for Eclipse in your patch names each jar individually. This seems more fragile: dependency upgrades will require updates to the dev-tools/eclipse/ setup. I think at least population of the library entries could be scripted, so that when the user runs ant eclipse, the configuration that's written out reflects the current state of the jars in the various lib/ directories. Any reason why scripting wouldn't work?

        Show
        Steve Rowe added a comment - here's the eclipse part (following the same conventions of your patch). basically for eclipse, getting things to work is just setting up the classpath and setting the whole project to UTF-8... but this takes a while even if you know everything you need to do. The IDEA configuration sets up dependencies as whole jar directories, but the setup for Eclipse in your patch names each jar individually. This seems more fragile: dependency upgrades will require updates to the dev-tools/eclipse/ setup. I think at least population of the library entries could be scripted, so that when the user runs ant eclipse , the configuration that's written out reflects the current state of the jars in the various lib/ directories. Any reason why scripting wouldn't work?
        Hide
        Robert Muir added a comment -

        here's the eclipse part (following the same conventions of your patch).

        basically for eclipse, getting things to work is just setting up the classpath and setting the whole project to UTF-8... but this takes a while even if you know everything you need to do.

        Show
        Robert Muir added a comment - here's the eclipse part (following the same conventions of your patch). basically for eclipse, getting things to work is just setting up the classpath and setting the whole project to UTF-8... but this takes a while even if you know everything you need to do.
        Hide
        Robert Muir added a comment -

        I don't have any experience with Eclipse, but I've read that Eclipse lacks hierarchical project layout, so although you could get a single build.xml to work, I don't think the multilayered Lucene/Solr build would work. Have you specifically tried with Lucene/Solr?

        Yes I have, but i (nor its automatic thingy) don't setup anything hierarchical. its just one big project with all the folders and libraries added to the build path.

        Show
        Robert Muir added a comment - I don't have any experience with Eclipse, but I've read that Eclipse lacks hierarchical project layout, so although you could get a single build.xml to work, I don't think the multilayered Lucene/Solr build would work. Have you specifically tried with Lucene/Solr? Yes I have, but i (nor its automatic thingy) don't setup anything hierarchical. its just one big project with all the folders and libraries added to the build path.
        Hide
        Steve Rowe added a comment - - edited

        True, but so can ant right? For example in eclipse i know there is a way to say 'from existing ant build file' (I have done this on accident and it setup all the classpaths etc by somehow examining our ant build)

        I don't have any experience with Eclipse, but I've read that Eclipse lacks hierarchical project layout, so although you could get a single build.xml to work, I don't think the multilayered Lucene/Solr build would work. Have you specifically tried with Lucene/Solr?

        And AFAIK, IntelliJ can't bootstrap from an Ant build, though IntelliJ can call tasks from an Ant build (the attached patch sets up this form of Ant integration).

        My question about LUCENE-2657 is I don't understand its relationship to this patch. Are you saying, if we had LUCENE-2657, then we do not need this patch at all?

        IntelliJ 9.0.4 doesn't understand all Maven build concepts. I haven't tried with IntelliJ 10.0.1 (the most recent version). I think we need this patch if we want full build and test functionality from IntelliJ. My concept of the relationship between the two is that this issue hosts the ideal IntelliJ configuration, and LUCENE-2657 hosts a Maven build which would ideally be usable by IntelliJ (and Eclipse) to bootstrap configuration, but it's not there (yet, and maybe never will be, not sure).

        Not understanding the syntax to either (they are both huge gobs of xml to me), this one appears much easier to maintain, because the LUCENE-2657 seems to have a lot of redundant/duplicated stuff such as system properties for running tests in every contrib module... or is it still very much a work in progress?

        I think the two issues are not mutually exclusive, at least not right now.

        LUCENE-2657 is very much a work in progress. My goals with LUCENE-2657 are (in order of importance):

        1. Enable people to build artifacts which can be used as dependencies in their projects, from non-released code: trunk and branch_3x.
        2. Provide a replacement for the current ant generate-maven-artifacts, for use in releases, or maybe just post-release maven repository population (I'm not sure how it's used right now).
        3. Allow Solr to be started from Maven using jetty-maven-plugin: SOLR-1218
        4. Provide an IDE configuration bootstrap source.

        Right now I'm working on building the Solr war, adding source and javadoc jar creation, and reducing configuration duplication, e.g. the system properties for running tests.

        Show
        Steve Rowe added a comment - - edited True, but so can ant right? For example in eclipse i know there is a way to say 'from existing ant build file' (I have done this on accident and it setup all the classpaths etc by somehow examining our ant build) I don't have any experience with Eclipse, but I've read that Eclipse lacks hierarchical project layout, so although you could get a single build.xml to work, I don't think the multilayered Lucene/Solr build would work. Have you specifically tried with Lucene/Solr? And AFAIK, IntelliJ can't bootstrap from an Ant build, though IntelliJ can call tasks from an Ant build (the attached patch sets up this form of Ant integration). My question about LUCENE-2657 is I don't understand its relationship to this patch. Are you saying, if we had LUCENE-2657 , then we do not need this patch at all? IntelliJ 9.0.4 doesn't understand all Maven build concepts. I haven't tried with IntelliJ 10.0.1 (the most recent version). I think we need this patch if we want full build and test functionality from IntelliJ. My concept of the relationship between the two is that this issue hosts the ideal IntelliJ configuration, and LUCENE-2657 hosts a Maven build which would ideally be usable by IntelliJ (and Eclipse) to bootstrap configuration, but it's not there (yet, and maybe never will be, not sure). Not understanding the syntax to either (they are both huge gobs of xml to me), this one appears much easier to maintain, because the LUCENE-2657 seems to have a lot of redundant/duplicated stuff such as system properties for running tests in every contrib module... or is it still very much a work in progress? I think the two issues are not mutually exclusive, at least not right now. LUCENE-2657 is very much a work in progress. My goals with LUCENE-2657 are (in order of importance): Enable people to build artifacts which can be used as dependencies in their projects, from non-released code: trunk and branch_3x. Provide a replacement for the current ant generate-maven-artifacts , for use in releases, or maybe just post-release maven repository population (I'm not sure how it's used right now). Allow Solr to be started from Maven using jetty-maven-plugin : SOLR-1218 Provide an IDE configuration bootstrap source. Right now I'm working on building the Solr war, adding source and javadoc jar creation, and reducing configuration duplication, e.g. the system properties for running tests.
        Hide
        Erick Erickson added a comment -

        FWIW, I used this patch for IntelliJ a few weeks ago and it was pretty smooth. Let me know if you need me to give it a whirl on either a Mac or Windows box.

        Show
        Erick Erickson added a comment - FWIW, I used this patch for IntelliJ a few weeks ago and it was pretty smooth. Let me know if you need me to give it a whirl on either a Mac or Windows box.
        Hide
        Robert Muir added a comment -

        Maven POMs can be used by both to bootstrap projects: LUCENE-2657. I've never tried with Eclipse, though. And IntelliJ doesn't grok everything, e.g. multiple content roots.

        True, but so can ant right? For example in eclipse i know there is a way to say 'from existing ant build file'
        (I have done this on accident and it setup all the classpaths etc by somehow examining our ant build)

        My question about LUCENE-2657 is I don't understand its relationship to this patch. Are you saying, if
        we had LUCENE-2657, then we do not need this patch at all?

        Not understanding the syntax to either (they are both huge gobs of xml to me), this one appears much
        easier to maintain, because the LUCENE-2657 seems to have a lot of redundant/duplicated stuff such
        as system properties for running tests in every contrib module... or is it still very much a work in progress?

        Show
        Robert Muir added a comment - Maven POMs can be used by both to bootstrap projects: LUCENE-2657 . I've never tried with Eclipse, though. And IntelliJ doesn't grok everything, e.g. multiple content roots. True, but so can ant right? For example in eclipse i know there is a way to say 'from existing ant build file' (I have done this on accident and it setup all the classpaths etc by somehow examining our ant build) My question about LUCENE-2657 is I don't understand its relationship to this patch. Are you saying, if we had LUCENE-2657 , then we do not need this patch at all? Not understanding the syntax to either (they are both huge gobs of xml to me), this one appears much easier to maintain, because the LUCENE-2657 seems to have a lot of redundant/duplicated stuff such as system properties for running tests in every contrib module... or is it still very much a work in progress?
        Hide
        Steve Rowe added a comment - - edited

        I don't think we should necessarily exclude checking in the project files into svn either, if thats what it takes. it seems this would be lesser than the evil we have now.

        I set up this patch to copy in the IntelliJ configuration files from another location (dev-tools/idea/), and svn:ignore the in-place config files, because each user will add/subtract stuff to the configuration – if the in-place config files were committed to the Lucene repo, every svn update would require merging with the remote version, and every svn commit would put the current state of the committer's config files into the repo. Both of those are bad, and both are avoided with my approach.

        Can we figure out some plan to make it really easy to get going in these two IDEs, with minimal maintenance?

        Maven POMs can be used by both to bootstrap projects: LUCENE-2657. I've never tried with Eclipse, though. And IntelliJ doesn't grok everything, e.g. multiple content roots.

        For intellij, how hard would be it to maintain this support if its committed? As a theoretical example, what would need to be done if we were to factor out the function queries from lucene and solr and combine them into a new module that solr depends on (and perhaps the lucene queryparsers contrib would depend on it two with some parsing support) ?

        I went through the process of adding a new solr contrib module here when you added analysis-extras. The effort was fairly small, maybe 2-3 hours, including copy/pasting/modifying an existing contrib's configuration and adding it to the project config in various places, then running all defined module tests. I think most of this could be scripted, which would seriously reduce the effort. I'll look at that the next time the code base is restructured. At a minimum, of course, there should be documentation on how to make changes.

        I guess with the concerns about maintenance, if nobody maintains the files then they arent using those IDEs and perhaps we could agree that if stuff like this got really out of date we would just delete it.

        +1. I think since my approach would have zero impact on non-users, this is the only potentially blocking concern.

        Show
        Steve Rowe added a comment - - edited I don't think we should necessarily exclude checking in the project files into svn either, if thats what it takes. it seems this would be lesser than the evil we have now. I set up this patch to copy in the IntelliJ configuration files from another location ( dev-tools/idea/ ), and svn:ignore the in-place config files, because each user will add/subtract stuff to the configuration – if the in-place config files were committed to the Lucene repo, every svn update would require merging with the remote version, and every svn commit would put the current state of the committer's config files into the repo. Both of those are bad, and both are avoided with my approach. Can we figure out some plan to make it really easy to get going in these two IDEs, with minimal maintenance? Maven POMs can be used by both to bootstrap projects: LUCENE-2657 . I've never tried with Eclipse, though. And IntelliJ doesn't grok everything, e.g. multiple content roots. For intellij, how hard would be it to maintain this support if its committed? As a theoretical example, what would need to be done if we were to factor out the function queries from lucene and solr and combine them into a new module that solr depends on (and perhaps the lucene queryparsers contrib would depend on it two with some parsing support) ? I went through the process of adding a new solr contrib module here when you added analysis-extras . The effort was fairly small, maybe 2-3 hours, including copy/pasting/modifying an existing contrib's configuration and adding it to the project config in various places, then running all defined module tests. I think most of this could be scripted, which would seriously reduce the effort. I'll look at that the next time the code base is restructured. At a minimum, of course, there should be documentation on how to make changes. I guess with the concerns about maintenance, if nobody maintains the files then they arent using those IDEs and perhaps we could agree that if stuff like this got really out of date we would just delete it. +1. I think since my approach would have zero impact on non-users, this is the only potentially blocking concern.
        Hide
        Robert Muir added a comment -

        Just a few thoughts/general questions:

        1. First of all, I think we really need to pursue making it easier to setup your development environment to attract more contributors. I personally find it difficult to use an IDE (at the moment I use 'ant test' but edit code with eclipse's editor, so that i get syntax completion etc). But at the moment setting up something that works is too high of a barrier to entry, it might sound extreme to say this, but I think we are basically discouraging contributors with the complexity.
        2. As far as making it easier for any particular IDE, my opinion is: as long as someone maintains it. But If i (or someone else) want to go re-organize the whole build and the consensus is that its a good change, I think the only thing that must work is the ant build. So the danger is that we will only have non-functional out-of-date IDE support... we don't want to slow down development either by requiring someone to fix ant, maven, eclipse, intellij, netbeans emacs, make, .vimrc, whatever. But on the other hand, what we have now is 'not working' in my opinion.
        3. I think (just off the top of my head) that a lot of new contributors will likely be using intellij or eclipse, and that these are the two big ones. I know uwe uses notepad and mike uses emacs, but i think these seem to be the most popular. I'm not trying to say we have to show bias/preference towards these two, but I think we need to address the reality and lower the barrier to entry. Can we figure out some plan to make it really easy to get going in these two IDEs, with minimal maintenance? I don't think we should necessarily exclude checking in the project files into svn either, if thats what it takes. it seems this would be lesser than the evil we have now. And of course, an alternative might be to do what this patch does, and also make a similar patch for eclipse too.
        4. For intellij, how hard would be it to maintain this support if its committed? As a theoretical example, what would need to be done if we were to factor out the function queries from lucene and solr and combine them into a new module that solr depends on (and perhaps the lucene queryparsers contrib would depend on it two with some parsing support) ?
        5. I guess with the concerns about maintenance, if nobody maintains the files then they arent using those IDEs and perhaps we could agree that if stuff like this got really out of date we would just delete it.

        Anyway, I just wanted to voice my support for the idea and get some discussion going as I would really like to see this situation simplified sooner than later.

        Show
        Robert Muir added a comment - Just a few thoughts/general questions: First of all, I think we really need to pursue making it easier to setup your development environment to attract more contributors. I personally find it difficult to use an IDE (at the moment I use 'ant test' but edit code with eclipse's editor, so that i get syntax completion etc). But at the moment setting up something that works is too high of a barrier to entry, it might sound extreme to say this, but I think we are basically discouraging contributors with the complexity. As far as making it easier for any particular IDE, my opinion is: as long as someone maintains it. But If i (or someone else) want to go re-organize the whole build and the consensus is that its a good change, I think the only thing that must work is the ant build. So the danger is that we will only have non-functional out-of-date IDE support... we don't want to slow down development either by requiring someone to fix ant, maven, eclipse, intellij, netbeans emacs, make, .vimrc, whatever. But on the other hand, what we have now is 'not working' in my opinion. I think (just off the top of my head) that a lot of new contributors will likely be using intellij or eclipse, and that these are the two big ones. I know uwe uses notepad and mike uses emacs, but i think these seem to be the most popular. I'm not trying to say we have to show bias/preference towards these two, but I think we need to address the reality and lower the barrier to entry. Can we figure out some plan to make it really easy to get going in these two IDEs, with minimal maintenance? I don't think we should necessarily exclude checking in the project files into svn either, if thats what it takes. it seems this would be lesser than the evil we have now. And of course, an alternative might be to do what this patch does, and also make a similar patch for eclipse too. For intellij, how hard would be it to maintain this support if its committed? As a theoretical example, what would need to be done if we were to factor out the function queries from lucene and solr and combine them into a new module that solr depends on (and perhaps the lucene queryparsers contrib would depend on it two with some parsing support) ? I guess with the concerns about maintenance, if nobody maintains the files then they arent using those IDEs and perhaps we could agree that if stuff like this got really out of date we would just delete it. Anyway, I just wanted to voice my support for the idea and get some discussion going as I would really like to see this situation simplified sooner than later.
        Hide
        David Smiley added a comment -

        It turns out that IntelliJ was rewriting my $MODULE_DIR$/../../ paths to paths relative to a path variable I defined on my system, and that is intended behavior according to JetBrains. I removed the path variable... I can live without it after all, and that problem doesn't exist anymore.

        Show
        David Smiley added a comment - It turns out that IntelliJ was rewriting my $MODULE_DIR$/../../ paths to paths relative to a path variable I defined on my system, and that is intended behavior according to JetBrains. I removed the path variable... I can live without it after all, and that problem doesn't exist anymore.
        Hide
        Steve Rowe added a comment -

        bringing things up from scratch was trivial after "ant idea". Perhaps nobody has mentioned lately just how much of a time saver this really is, but I will right now... Removing a barrier to someone getting started is always a good thing.

        +1

        I've added a note on the How To Contribute page under IntelliJ setup about this FWIW.

        Thanks!

        Show
        Steve Rowe added a comment - bringing things up from scratch was trivial after "ant idea". Perhaps nobody has mentioned lately just how much of a time saver this really is, but I will right now... Removing a barrier to someone getting started is always a good thing. +1 I've added a note on the How To Contribute page under IntelliJ setup about this FWIW. Thanks!
        Hide
        Erick Erickson added a comment -

        Steven:

        Thanks, using UNIX2DOS made the patch apply cleanly, and bringing things up from scratch was trivial after "ant idea". Perhaps nobody has mentioned lately just how much of a time saver this really is, but I will right now... Removing a barrier to someone getting started is always a good thing.

        I've added a note on the How To Contribute page under IntelliJ setup about this FWIW.

        Erick

        Show
        Erick Erickson added a comment - Steven: Thanks, using UNIX2DOS made the patch apply cleanly, and bringing things up from scratch was trivial after "ant idea". Perhaps nobody has mentioned lately just how much of a time saver this really is, but I will right now... Removing a barrier to someone getting started is always a good thing. I've added a note on the How To Contribute page under IntelliJ setup about this FWIW. Erick
        Hide
        Steve Rowe added a comment -

        My IDEA project for Lucene has all .iml files in root dir. Various content-roots for sure. I recall that worked for at least several major IDEA versions.

        I have IntelliJ IDEA 9.0.4.

        I tried putting the ant contrib's .iml file in the lucene/ directory, alongside lucene.iml, so that output paths (to lucene/build/contrib/ant/...) wouldn't need "../" elements in them.

        IntelliJ complained, saying that two modules cannot share the same content root. "content root", as far as I can tell, is encoded implicitly as the directory in which the .iml file for the module is located.

        Maybe there's a way to trick IntelliJ into doing this, but I'm not sure how.

        Show
        Steve Rowe added a comment - My IDEA project for Lucene has all .iml files in root dir. Various content-roots for sure. I recall that worked for at least several major IDEA versions. I have IntelliJ IDEA 9.0.4. I tried putting the ant contrib's .iml file in the lucene/ directory, alongside lucene.iml , so that output paths (to lucene/build/contrib/ant/... ) wouldn't need "../" elements in them. IntelliJ complained, saying that two modules cannot share the same content root. "content root", as far as I can tell, is encoded implicitly as the directory in which the .iml file for the module is located. Maybe there's a way to trick IntelliJ into doing this, but I'm not sure how.
        Hide
        Earwin Burrfoot added a comment - - edited

        I wonder if several .iml files can be in the same directory but their so-called "content-roots" would be set to where they are now?

        I'm pretty sure IntelliJ allows only one .iml per directory.

        My IDEA project for Lucene has all .iml files in root dir. Various content-roots for sure.

        I recall that worked for at least several major IDEA versions.

        Show
        Earwin Burrfoot added a comment - - edited I wonder if several .iml files can be in the same directory but their so-called "content-roots" would be set to where they are now? I'm pretty sure IntelliJ allows only one .iml per directory. My IDEA project for Lucene has all .iml files in root dir. Various content-roots for sure. I recall that worked for at least several major IDEA versions.
        Hide
        Steve Rowe added a comment -

        The latest patch fails to apply on my Windows box ... Assertion failed: hunk, file ../patch-2.5.9-src/patch.c, line 354

        It says here that this problem is caused by mismatched line endings on Windows: http://www.mail-archive.com/gnuwin32-users@lists.sourceforge.net/msg01528.html

        Do you use Cygwin? I do, and I don't see any problems patching with the latest patches using /usr/bin/patch from a Bash prompt on Windows 7. I do see this message after each file is patched:

        (Stripping trailing CRs from patch.)

        So maybe you could either switch to using Cygwin's patch, or use dos2unix or unix2dos (depending on which flavor is required for the executable you've got), or something like this perl script if you don't have dos2unix/unix2dos:

        perl -0777 -pi.bak -e 's/\r\n/\n/g' LUCENE-2611.patch
        
        Show
        Steve Rowe added a comment - The latest patch fails to apply on my Windows box ... Assertion failed: hunk, file ../patch-2.5.9-src/patch.c, line 354 It says here that this problem is caused by mismatched line endings on Windows: http://www.mail-archive.com/gnuwin32-users@lists.sourceforge.net/msg01528.html Do you use Cygwin? I do, and I don't see any problems patching with the latest patches using /usr/bin/patch from a Bash prompt on Windows 7. I do see this message after each file is patched: (Stripping trailing CRs from patch.) So maybe you could either switch to using Cygwin's patch, or use dos2unix or unix2dos (depending on which flavor is required for the executable you've got), or something like this perl script if you don't have dos2unix/unix2dos: perl -0777 -pi.bak -e 's/\r\n/\n/g' LUCENE-2611.patch
        Hide
        Steve Rowe added a comment -

        The input .iml file that comes from the patch has a path like so for the benchmark contrib module (as an example):
        <output url="file://$MODULE_DIR$/../../build/contrib/benchmark/classes/java" />
        And after IntelliJ reads this project and saves its files, this is re-written to:
        <output url="file://$SmileyDev$/Projects/Projects-External/lucene-solr_trunk/lucene/build/contrib/benchmark/classes/java" />
        Obviously the path as given in the patch could be simplified to not needlessly use "../" and I'm pretty sure then that IntelliJ will not rewrite it.

        Not sure why we see different behavior, but on my systems (Windows 7 and Vista), IntelliJ does not rewrite to absolute paths. In fact, IntelliJ will generate the $MODULE_DIR$/../ style output paths when you create new modules that output to non-descendant directories.

        I wonder if several .iml files can be in the same directory but their so-called "content-roots" would be set to where they are now?

        I'm pretty sure IntelliJ allows only one .iml per directory.

        Or, do away with module-specific build output directories and have them inherit the project level. Yes, this means the output directory is then inconsistent with ant. That matters little to me, but I can understand others having a difference of opinion..

        I want to keep the output directories the same - I often switch between IntelliJ (for quick turnaround dev/testing) and Ant (pre-JIRA-patch testing).

        Show
        Steve Rowe added a comment - The input .iml file that comes from the patch has a path like so for the benchmark contrib module (as an example): <output url="file://$MODULE_DIR$/../../build/contrib/benchmark/classes/java" /> And after IntelliJ reads this project and saves its files, this is re-written to: <output url="file://$SmileyDev$/Projects/Projects-External/lucene-solr_trunk/lucene/build/contrib/benchmark/classes/java" /> Obviously the path as given in the patch could be simplified to not needlessly use "../" and I'm pretty sure then that IntelliJ will not rewrite it. Not sure why we see different behavior, but on my systems (Windows 7 and Vista), IntelliJ does not rewrite to absolute paths. In fact, IntelliJ will generate the $MODULE_DIR$/../ style output paths when you create new modules that output to non-descendant directories. I wonder if several .iml files can be in the same directory but their so-called "content-roots" would be set to where they are now? I'm pretty sure IntelliJ allows only one .iml per directory. Or, do away with module-specific build output directories and have them inherit the project level. Yes, this means the output directory is then inconsistent with ant. That matters little to me, but I can understand others having a difference of opinion.. I want to keep the output directories the same - I often switch between IntelliJ (for quick turnaround dev/testing) and Ant (pre-JIRA-patch testing).
        Hide
        Erick Erickson added a comment -

        Steven:

        I love Windows. I mean, I really LOVE windows.... The latest patch fails to apply on my Windows box with the message below, but works magnificently on my Mac.....

        patching file dev-tools/idea/lucene/lucene.iml
        Assertion failed: hunk, file ../patch-2.5.9-src/patch.c, line 354

        What it means I don't know, but I thought I'd mention it...

        Erick

        Show
        Erick Erickson added a comment - Steven: I love Windows. I mean, I really LOVE windows.... The latest patch fails to apply on my Windows box with the message below, but works magnificently on my Mac..... patching file dev-tools/idea/lucene/lucene.iml Assertion failed: hunk, file ../patch-2.5.9-src/patch.c, line 354 What it means I don't know, but I thought I'd mention it... Erick
        Hide
        David Smiley added a comment -

        Thanks for the improvement Steve. FWIW I reported this problem to IntelliJ http://youtrack.jetbrains.net/issue/IDEA-62977 The only alternative I can think of is.... I wonder if several .iml files can be in the same directory but their so-called "content-roots" would be set to where they are now? I dunno. Or, do away with module-specific build output directories and have them inherit the project level. Yes, this means the output directory is then inconsistent with ant. That matters little to me, but I can understand others having a difference of opinion..

        Show
        David Smiley added a comment - Thanks for the improvement Steve. FWIW I reported this problem to IntelliJ http://youtrack.jetbrains.net/issue/IDEA-62977 The only alternative I can think of is.... I wonder if several .iml files can be in the same directory but their so-called "content-roots" would be set to where they are now? I dunno. Or, do away with module-specific build output directories and have them inherit the project level. Yes, this means the output directory is then inconsistent with ant. That matters little to me, but I can understand others having a difference of opinion..
        Hide
        Steve Rowe added a comment -

        Both the trunk and the branch_3x patches add a new goal to the top-level build.xml: clean-idea. This new goal removes all IntelliJ IDEA configuration files. I run it before I run ant idea, to start from a clean slate.

        Show
        Steve Rowe added a comment - Both the trunk and the branch_3x patches add a new goal to the top-level build.xml : clean-idea . This new goal removes all IntelliJ IDEA configuration files. I run it before I run ant idea , to start from a clean slate.
        Hide
        Steve Rowe added a comment -

        Two fixes for trunk patch:

        1. In .idea/compiler.xml, added *.gz to the list of resource file patterns to copy to the build directory, so that europarl.lines.txt.gz makes it there.
        2. Renamed .idea/libraries/Servlet_API_24.xml to .idea/libraries/Servlet_API_2_4.xml; IntelliJ otherwise renames the file (because of the library name "Servlet API 2.4", where non-alphanum chars are apparently converted to underscores), and then successive calls to "ant idea" will create extra copies of the library file in .idea/libraries, with slightly different names....
        Show
        Steve Rowe added a comment - Two fixes for trunk patch: In .idea/compiler.xml , added *.gz to the list of resource file patterns to copy to the build directory, so that europarl.lines.txt.gz makes it there. Renamed .idea/libraries/Servlet_API_24.xml to .idea/libraries/Servlet_API_2_4.xml ; IntelliJ otherwise renames the file (because of the library name "Servlet API 2.4", where non-alphanum chars are apparently converted to underscores), and then successive calls to "ant idea" will create extra copies of the library file in .idea/libraries , with slightly different names....
        Hide
        Steve Rowe added a comment -

        3.x branch patch brought up-to-date, with shared module libraries moved to instead be project libraries

        Show
        Steve Rowe added a comment - 3.x branch patch brought up-to-date, with shared module libraries moved to instead be project libraries
        Hide
        Steve Rowe added a comment -

        Re-added with ASF license grant

        Show
        Steve Rowe added a comment - Re-added with ASF license grant
        Hide
        Steve Rowe added a comment -

        Patch for latest trunk.

        David, thanks for your input. I tried to get rid of the "../" path components, but couldn't - IntelliJ didn't like the use of $PROJECT_DIR$ in .iml files. Do you have any suggestions?

        Also, I moved all shared library directories to be project libraries (e.g. JUnit), rather than defined as module libraries in each module that uses them.

        Show
        Steve Rowe added a comment - Patch for latest trunk. David, thanks for your input. I tried to get rid of the "../" path components, but couldn't - IntelliJ didn't like the use of $PROJECT_DIR$ in .iml files. Do you have any suggestions? Also, I moved all shared library directories to be project libraries (e.g. JUnit), rather than defined as module libraries in each module that uses them.
        Hide
        David Smiley added a comment -

        I used one of the attached patch files (I forget which at this point; it's confusing which is the right one) and it's been working great until today. I noticed today that my .iml files have absolute machine paths to the output classes and output test classes, even though they should be within the $MODULE_DIR$ path variable, they are not. Consequently, if I move or copy my lucene-solr checkout (which I did), the paths become wrong.

        The input .iml file that comes from the patch has a path like so for the benchmark contrib module (as an example):
        <output url="file://$MODULE_DIR$/../../build/contrib/benchmark/classes/java" />
        And after IntelliJ reads this project and saves its files, this is re-written to:
        <output url="file://$SmileyDev$/Projects/Projects-External/lucene-solr_trunk/lucene/build/contrib/benchmark/classes/java" />
        Obviously the path as given in the patch could be simplified to not needlessly use "../" and I'm pretty sure then that IntelliJ will not rewrite it.

        There's a little different problem for the dependencies. The Junit dependency came from the patch like so:
        <root url="jar://$MODULE_DIR$/../../lib/junit-4.7.jar!/" />
        And as you might guess, this was rewritten to an absolute path, unfortunately. I think fixing this the right way is to create project level libraries for Junit and some others, and then simply reference those libraries from the .iml files. That is certainly best practice, and changing the version number of junit won't invalidate all these .iml files.

        Show
        David Smiley added a comment - I used one of the attached patch files (I forget which at this point; it's confusing which is the right one) and it's been working great until today. I noticed today that my .iml files have absolute machine paths to the output classes and output test classes, even though they should be within the $MODULE_DIR$ path variable, they are not. Consequently, if I move or copy my lucene-solr checkout (which I did), the paths become wrong. The input .iml file that comes from the patch has a path like so for the benchmark contrib module (as an example): <output url="file://$MODULE_DIR$/../../build/contrib/benchmark/classes/java" /> And after IntelliJ reads this project and saves its files, this is re-written to: <output url="file://$SmileyDev$/Projects/Projects-External/lucene-solr_trunk/lucene/build/contrib/benchmark/classes/java" /> Obviously the path as given in the patch could be simplified to not needlessly use "../" and I'm pretty sure then that IntelliJ will not rewrite it. There's a little different problem for the dependencies. The Junit dependency came from the patch like so: <root url="jar://$MODULE_DIR$/../../lib/junit-4.7.jar!/" /> And as you might guess, this was rewritten to an absolute path, unfortunately. I think fixing this the right way is to create project level libraries for Junit and some others, and then simply reference those libraries from the .iml files. That is certainly best practice, and changing the version number of junit won't invalidate all these .iml files.
        Hide
        Steve Rowe added a comment -

        Once Robert's latest patch on SOLR-2002 gets applied – it moves around some of the Solr module structure – the IntelliJ setup patches will need to be adjusted.

        Show
        Steve Rowe added a comment - Once Robert's latest patch on SOLR-2002 gets applied – it moves around some of the Solr module structure – the IntelliJ setup patches will need to be adjusted.
        Hide
        Steve Rowe added a comment -

        So the only problem is the directory may not exist... and I think a static {} mkdirs() in both classes will fix this easily.

        You are right - I ran all the trunk and 3.X branch modules' test runs under IntelliJ using the static{} mkdirs() fix you checked in, and all tests passed.

        seems like we are making progress!

        Yes, we are. Thanks very much for your help, Robert.

        I don't like the case #3 Lucene tests above that 're-use' the same named directory every time, its just asking for trouble. these should do like the rest of the lucene/solr tests and use a "real" tempdir thats got a random name just to be safe... this is a separate problem.

        +1

        Show
        Steve Rowe added a comment - So the only problem is the directory may not exist... and I think a static {} mkdirs() in both classes will fix this easily. You are right - I ran all the trunk and 3.X branch modules' test runs under IntelliJ using the static{} mkdirs() fix you checked in, and all tests passed. seems like we are making progress! Yes, we are. Thanks very much for your help, Robert. I don't like the case #3 Lucene tests above that 're-use' the same named directory every time, its just asking for trouble. these should do like the rest of the lucene/solr tests and use a "real" tempdir thats got a random name just to be safe... this is a separate problem. +1
        Hide
        Robert Muir added a comment -

        OK, I added the mkdirs() in 993199/993200 (3x).

        seems like we are making progress!

        Show
        Robert Muir added a comment - OK, I added the mkdirs() in 993199/993200 (3x). seems like we are making progress!
        Hide
        Robert Muir added a comment -

        ok, here is all we need for the mkdirs(), as LuceneTestCase.TEMP_DIR refers to the LuceneTestCaseJ4 it should be created no matter what.

        Show
        Robert Muir added a comment - ok, here is all we need for the mkdirs(), as LuceneTestCase.TEMP_DIR refers to the LuceneTestCaseJ4 it should be created no matter what.
        Hide
        Robert Muir added a comment -

        2. All modules' test runs now use the same tempDir sysprop as the Ant tests.

        OK, sorry I missed that (I apologize, i'm not able to make much sense of the patch since i dont understand IntelliJ).

        So the only problem is the directory may not exist... and I think a static {} mkdirs() in both classes will fix this easily.

        But still, I don't like the case #3 Lucene tests above that 're-use' the same named directory every time, its just asking for trouble. these should do like the rest of the lucene/solr tests and use a "real" tempdir thats got a random name just to be safe... this is a separate problem.

        Show
        Robert Muir added a comment - 2. All modules' test runs now use the same tempDir sysprop as the Ant tests. OK, sorry I missed that (I apologize, i'm not able to make much sense of the patch since i dont understand IntelliJ). So the only problem is the directory may not exist... and I think a static {} mkdirs() in both classes will fix this easily. But still, I don't like the case #3 Lucene tests above that 're-use' the same named directory every time, its just asking for trouble. these should do like the rest of the lucene/solr tests and use a "real" tempdir thats got a random name just to be safe... this is a separate problem.
        Hide
        Steve Rowe added a comment -

        Ok, I fixed the Abstract part for the clustering tests in revisions 993194/993195 (3x)

        Cool, thanks

        It's important to isolate the temporary directory used by 3.X branch tests from that used by trunk, because otherwise 3.X branch tests will fail with unknown index version errors.

        Just another thought: I don't know intelliJ at all, but is it possible when running tests to set the -DtempDir sysprop for IntelliJ to a directory in "build/" or similar? maybe this would be better than defaulting to the system default tempdir if its not set, but we still need to address the issues you brought up too (at least for eclipse users)

        Yes, I actually did exactly what you suggest - as I said above:

        2. All modules' test runs now use the same tempDir sysprop as the Ant tests.

        Show
        Steve Rowe added a comment - Ok, I fixed the Abstract part for the clustering tests in revisions 993194/993195 (3x) Cool, thanks It's important to isolate the temporary directory used by 3.X branch tests from that used by trunk, because otherwise 3.X branch tests will fail with unknown index version errors. Just another thought: I don't know intelliJ at all, but is it possible when running tests to set the -DtempDir sysprop for IntelliJ to a directory in "build/" or similar? maybe this would be better than defaulting to the system default tempdir if its not set, but we still need to address the issues you brought up too (at least for eclipse users) Yes, I actually did exactly what you suggest - as I said above: 2. All modules' test runs now use the same tempDir sysprop as the Ant tests.
        Hide
        Robert Muir added a comment -

        Ok, I fixed the Abstract part for the clustering tests in revisions 993194/993195 (3x)

        It's important to isolate the temporary directory used by 3.X branch tests from that used by trunk, because otherwise 3.X branch tests will fail with unknown index version errors.

        Just another thought: I don't know intelliJ at all, but is it possible when running tests to set the -DtempDir sysprop for IntelliJ to a directory in "build/" or similar? maybe this would be better than defaulting to the system default tempdir if its not set, but we still need to address the issues you brought up too (at least for eclipse users)

        Show
        Robert Muir added a comment - Ok, I fixed the Abstract part for the clustering tests in revisions 993194/993195 (3x) It's important to isolate the temporary directory used by 3.X branch tests from that used by trunk, because otherwise 3.X branch tests will fail with unknown index version errors. Just another thought: I don't know intelliJ at all, but is it possible when running tests to set the -DtempDir sysprop for IntelliJ to a directory in "build/" or similar? maybe this would be better than defaulting to the system default tempdir if its not set, but we still need to address the issues you brought up too (at least for eclipse users)
        Hide
        Robert Muir added a comment -

        AbstractClusteringTest is made abstract, to stop IntelliJ from failing because it has no test methods.

        We should fix this. I think we should also fix the name (*TestBase or similar) for ant.

        TEMP_DIR.mkdirs() is added to setUp() in LuceneTestCase and LuceneTestCaseJ4, so that various tests don't fail because the parent temporary directory doesn't exist.

        I don't think this should occur in setUp(). Ant makes this up front per-jvm, but for IDE's we should do it in a static { } block, then its consistent with ant.

        It's important to isolate the temporary directory used by 3.X branch tests from that used by trunk, because otherwise 3.X branch tests will fail with unknown index version errors. (The Sun Oracle JDK's implementation of File.mkdirs() first checks for the existence of the directory before creating its parent directories, so I don't think this will slow Ant testing down much.)

        I completely agree, but I don't think we should do it in setUp to address it. Currently there are:

        1. Lucene Tests that use newDirectory(): these get a TEMP_DIR + random subdir
        2. Solr tests: these get a TEMP_DIR + random subdir
        3. Lucene Tests that don't use newDirectory, but use a hardcoded TEMP_DIR + path. we should fix these to use a random subdir.

        The issue with #3 being, that if you aren't using ANT, the TEMP_DIR defaults to your OS system temp directory, and if you have different checkouts of lucene they will share it... so we should fix these tests... they shouldnt be hard to find.

        Show
        Robert Muir added a comment - AbstractClusteringTest is made abstract, to stop IntelliJ from failing because it has no test methods. We should fix this. I think we should also fix the name (*TestBase or similar) for ant. TEMP_DIR.mkdirs() is added to setUp() in LuceneTestCase and LuceneTestCaseJ4, so that various tests don't fail because the parent temporary directory doesn't exist. I don't think this should occur in setUp(). Ant makes this up front per-jvm, but for IDE's we should do it in a static { } block, then its consistent with ant. It's important to isolate the temporary directory used by 3.X branch tests from that used by trunk, because otherwise 3.X branch tests will fail with unknown index version errors. (The Sun Oracle JDK's implementation of File.mkdirs() first checks for the existence of the directory before creating its parent directories, so I don't think this will slow Ant testing down much.) I completely agree, but I don't think we should do it in setUp to address it. Currently there are: Lucene Tests that use newDirectory(): these get a TEMP_DIR + random subdir Solr tests: these get a TEMP_DIR + random subdir Lucene Tests that don't use newDirectory, but use a hardcoded TEMP_DIR + path. we should fix these to use a random subdir. The issue with #3 being, that if you aren't using ANT, the TEMP_DIR defaults to your OS system temp directory, and if you have different checkouts of lucene they will share it... so we should fix these tests... they shouldnt be hard to find.
        Hide
        Steve Rowe added a comment -

        Two other changes in this iteration of the two non-test patches that I forgot to mention:

        1. The Lucene {{contrib/db/bdb {,-je}

          }} jar dependency download tasks are now called from the idea task, so that the entire project can be built from IntelliJ out of the box.

        2. A message is now printed by ant idea telling the user to manually configure the Project SDK.
        Show
        Steve Rowe added a comment - Two other changes in this iteration of the two non-test patches that I forgot to mention: The Lucene {{contrib/db/bdb {,-je} }} jar dependency download tasks are now called from the idea task, so that the entire project can be built from IntelliJ out of the box. A message is now printed by ant idea telling the user to manually configure the Project SDK.
        Hide
        Steve Rowe added a comment -

        These patches allow all modules' tests to pass for me under IntelliJ - as long as no-one objects to these changes and/or finds any problems with them, I think they're all ready to commit.

        LUCENE-2611_test_2.patch changes two things (and can be applied both to trunk and the 3.X branch):

        1. AbstractClusteringTest is made abstract, to stop IntelliJ from failing because it has no test methods.
        2. TEMP_DIR.mkdirs() is added to setUp() in LuceneTestCase and LuceneTestCaseJ4, so that various tests don't fail because the parent temporary directory doesn't exist. It's important to isolate the temporary directory used by 3.X branch tests from that used by trunk, because otherwise 3.X branch tests will fail with unknown index version errors. (The Sun Oracle JDK's implementation of File.mkdirs() first checks for the existence of the directory before creating its parent directories, so I don't think this will slow Ant testing down much.)

        The {{LUCENE-2611

        {,-branch-3x}

        .patch}} files differ from the previous iteration in the following ways:

        1. The circular dependencies problem Erick noted are fixed:

          [...] on the project settings page, there are "circular dependencies"...
          1. queries, misc, common, remote
          2. solr, extraction

        2. All modules' test runs now use the same tempDir sysprop as the Ant tests.
        3. Removed the junit-mkdirs Ant pre-task for some modules' test runs, because it was not creating the correct directories to enable using build/.../test/ for {{LuceneTestCase {,J4}

          .TEMP_DIR}}.

        Show
        Steve Rowe added a comment - These patches allow all modules' tests to pass for me under IntelliJ - as long as no-one objects to these changes and/or finds any problems with them, I think they're all ready to commit. LUCENE-2611 _test_2.patch changes two things (and can be applied both to trunk and the 3.X branch): AbstractClusteringTest is made abstract, to stop IntelliJ from failing because it has no test methods. TEMP_DIR.mkdirs() is added to setUp() in LuceneTestCase and LuceneTestCaseJ4 , so that various tests don't fail because the parent temporary directory doesn't exist. It's important to isolate the temporary directory used by 3.X branch tests from that used by trunk, because otherwise 3.X branch tests will fail with unknown index version errors. (The Sun Oracle JDK's implementation of File.mkdirs() first checks for the existence of the directory before creating its parent directories, so I don't think this will slow Ant testing down much.) The {{ LUCENE-2611 {,-branch-3x} .patch}} files differ from the previous iteration in the following ways: The circular dependencies problem Erick noted are fixed: [...] on the project settings page, there are "circular dependencies"... 1. queries, misc, common, remote 2. solr, extraction All modules' test runs now use the same tempDir sysprop as the Ant tests. Removed the junit-mkdirs Ant pre-task for some modules' test runs, because it was not creating the correct directories to enable using build/.../test/ for {{LuceneTestCase {,J4} .TEMP_DIR}}.
        Hide
        Yonik Seeley added a comment -

        Sweet, nice job! The Solr tests now take less than 2 minutes to run for me!

        Show
        Yonik Seeley added a comment - Sweet, nice job! The Solr tests now take less than 2 minutes to run for me!
        Hide
        Robert Muir added a comment -

        The test patch could be profitably backported to the 3.X branch, couldn't it?

        i dunno about profitable, but yes: r992474

        Show
        Robert Muir added a comment - The test patch could be profitably backported to the 3.X branch, couldn't it? i dunno about profitable, but yes: r992474
        Hide
        Steve Rowe added a comment -

        The test patch could be profitably backported to the 3.X branch, couldn't it?

        Show
        Steve Rowe added a comment - The test patch could be profitably backported to the 3.X branch, couldn't it?
        Hide
        Robert Muir added a comment -

        OK I committed this (revision 992424). Let me know if you see other test problems with IntelliJ.

        Show
        Robert Muir added a comment - OK I committed this (revision 992424). Let me know if you see other test problems with IntelliJ.
        Hide
        Robert Muir added a comment -

        ok, the problem with TestBinaryField was really that a previous test (BadIndexSchemaTest) does some evil things and doesnt clear SolrConfig.severeErrors.

        additionally it made a wasted jetty, so i fixed this too.

        all tests pass with forking disabled, I plan to commit soon. this also shaves a good amount of time off solr's 'ant test'.

        Show
        Robert Muir added a comment - ok, the problem with TestBinaryField was really that a previous test (BadIndexSchemaTest) does some evil things and doesnt clear SolrConfig.severeErrors. additionally it made a wasted jetty, so i fixed this too. all tests pass with forking disabled, I plan to commit soon. this also shaves a good amount of time off solr's 'ant test'.
        Hide
        Steve Rowe added a comment -

        ok, i think for eclipse we had a similar problem, fixed by making them 'abstract'. this seems good to do though, are there any other base test classes that need the @Ignore too?

        I'm pretty sure making these classes abstract would work for IntelliJ too - I just thought that @Ignore was a less invasive change. I didn't come across any other classes that need this treatment.

        Show
        Steve Rowe added a comment - ok, i think for eclipse we had a similar problem, fixed by making them 'abstract'. this seems good to do though, are there any other base test classes that need the @Ignore too? I'm pretty sure making these classes abstract would work for IntelliJ too - I just thought that @Ignore was a less invasive change. I didn't come across any other classes that need this treatment.
        Hide
        Robert Muir added a comment -

        This patch fixes TestVariableResolver (problem is that static EvaluatorBag.dateMathParser remembers its default time zone from a previous test case

        one mystery solved! now only TestBinaryField is left, and I think we can turn off this forking.

        and annotates SolrTestCaseJ4 and LuceneTestCaseJ4 with @Ignore, since IntelliJ can neither select only @Test-annoted tests nor select tests matching patterns Test*.java,*Test.java - it just looks for classes extending TestCase and fails the test case when no test methods are found.

        ok, i think for eclipse we had a similar problem, fixed by making them 'abstract'. this seems good to do though, are there any other base test classes that need the @Ignore too?

        Show
        Robert Muir added a comment - This patch fixes TestVariableResolver (problem is that static EvaluatorBag.dateMathParser remembers its default time zone from a previous test case one mystery solved! now only TestBinaryField is left, and I think we can turn off this forking. and annotates SolrTestCaseJ4 and LuceneTestCaseJ4 with @Ignore, since IntelliJ can neither select only @Test-annoted tests nor select tests matching patterns Test*.java,*Test.java - it just looks for classes extending TestCase and fails the test case when no test methods are found. ok, i think for eclipse we had a similar problem, fixed by making them 'abstract'. this seems good to do though, are there any other base test classes that need the @Ignore too?
        Hide
        Steve Rowe added a comment -

        These patches add "-ea" to the lucene module test run configuration's VM parameters, and add a test run configuration for the extras module (under the dataimporthandler module).

        Show
        Steve Rowe added a comment - These patches add "-ea" to the lucene module test run configuration's VM parameters, and add a test run configuration for the extras module (under the dataimporthandler module).
        Hide
        Steve Rowe added a comment -

        In DIH's TestVariableResolver, two tests use the same code and fail in the same way:

        yeah, that one and TestBinaryField i havent figured out yet !

        This patch fixes TestVariableResolver (problem is that static EvaluatorBag.dateMathParser remembers its default time zone from a previous test case - solution is to reset it to the same default time zone as the current test case), and annotates SolrTestCaseJ4 and LuceneTestCaseJ4 with @Ignore, since IntelliJ can neither select only @Test-annoted tests nor select tests matching patterns Test*.java,*Test.java - it just looks for classes extending TestCase and fails the test case when no test methods are found.

        Show
        Steve Rowe added a comment - In DIH's TestVariableResolver, two tests use the same code and fail in the same way: yeah, that one and TestBinaryField i havent figured out yet ! This patch fixes TestVariableResolver (problem is that static EvaluatorBag.dateMathParser remembers its default time zone from a previous test case - solution is to reset it to the same default time zone as the current test case), and annotates SolrTestCaseJ4 and LuceneTestCaseJ4 with @Ignore , since IntelliJ can neither select only @Test -annoted tests nor select tests matching patterns Test*.java,*Test.java - it just looks for classes extending TestCase and fails the test case when no test methods are found.
        Hide
        Robert Muir added a comment -

        Three Lucene TestIndexWriter tests fail because expected exceptions don't happen: testExceptionDocumentsWriter, testExceptionOnMergeInit, and testRollbackExceptionHang.

        you need to enable assertions (-ea) when running junit tests, then these should pass.

        In DIH's TestVariableResolver, two tests use the same code and fail in the same way:

        yeah, that one and TestBinaryField i havent figured out yet !

        Show
        Robert Muir added a comment - Three Lucene TestIndexWriter tests fail because expected exceptions don't happen: testExceptionDocumentsWriter, testExceptionOnMergeInit, and testRollbackExceptionHang. you need to enable assertions (-ea) when running junit tests, then these should pass. In DIH's TestVariableResolver, two tests use the same code and fail in the same way: yeah, that one and TestBinaryField i havent figured out yet !
        Hide
        Steve Rowe added a comment -

        Steven, I made it further and cleared up most fails.

        Cool! Fewer failures for me on IntelliJ with this patch. Thanks for adding sysprop clearing to base tests.

        I'm still seeing Solr network issues, likely a firewall issue.

        Three Lucene TestIndexWriter tests fail because expected exceptions don't happen: testExceptionDocumentsWriter, testExceptionOnMergeInit, and testRollbackExceptionHang.

        In DIH's TestVariableResolver, two tests use the same code and fail in the same way:

        Assert.assertEquals(new SimpleDateFormat("yyyy-MM-dd HH:mm").format(dmp.parseMath("/DAY")), s);


        org.junit.ComparisonFailure:
        Expected :2010-08-31 00:00
        Actual :2010-08-30 15:00

        But when I re-run TestVariableResolver alone, all tests succeed. I suspect there is a similar sysprop leakage happening here, but I haven't tracked it down yet.

        Show
        Steve Rowe added a comment - Steven, I made it further and cleared up most fails. Cool! Fewer failures for me on IntelliJ with this patch. Thanks for adding sysprop clearing to base tests. I'm still seeing Solr network issues, likely a firewall issue. Three Lucene TestIndexWriter tests fail because expected exceptions don't happen: testExceptionDocumentsWriter, testExceptionOnMergeInit, and testRollbackExceptionHang. In DIH's TestVariableResolver, two tests use the same code and fail in the same way: Assert.assertEquals( new SimpleDateFormat( "yyyy-MM-dd HH:mm" ).format(dmp.parseMath( "/DAY" )), s); org.junit.ComparisonFailure: Expected :2010-08-31 00:00 Actual :2010-08-30 15:00 But when I re-run TestVariableResolver alone, all tests succeed. I suspect there is a similar sysprop leakage happening here, but I haven't tracked it down yet.
        Hide
        Robert Muir added a comment -

        Steven, I made it further and cleared up most fails.
        I've got 2 test fails now total, probably some statics or sysprops somewhere.

        I turned of test forking in the patch to make these easier to find from 'ant'

        Show
        Robert Muir added a comment - Steven, I made it further and cleared up most fails. I've got 2 test fails now total, probably some statics or sysprops somewhere. I turned of test forking in the patch to make these easier to find from 'ant'
        Hide
        Robert Muir added a comment -

        Sounds good. Thanks for your help.

        No prob, no promises i can get that working, but ideally solr tests would run without forking, like lucene tests.
        when we did this for lucene it cut the time down significantly... i can just turn on forkMode=perBatch and see the issues you see I think.

        Show
        Robert Muir added a comment - Sounds good. Thanks for your help. No prob, no promises i can get that working, but ideally solr tests would run without forking, like lucene tests. when we did this for lucene it cut the time down significantly... i can just turn on forkMode=perBatch and see the issues you see I think.
        Hide
        Steve Rowe added a comment -

        I don't think this is the same problem? This is just ensuring the class extends SolrTestCaseJ4 and doesnt have a 'nested test', there shouldn't be any others with the same problem as this test.

        Then we can separately address your solr.solr.home problem within the base classes as you suggest, this is a separate problem I think.

        Sounds good. Thanks for your help.

        Show
        Steve Rowe added a comment - I don't think this is the same problem? This is just ensuring the class extends SolrTestCaseJ4 and doesnt have a 'nested test', there shouldn't be any others with the same problem as this test. Then we can separately address your solr.solr.home problem within the base classes as you suggest, this is a separate problem I think. Sounds good. Thanks for your help.
        Hide
        Robert Muir added a comment -

        There are other tests in Solr that have the same problem (solr.solr.home cross-test environment pollution)

        I don't think this is the same problem? This is just ensuring the class extends SolrTestCaseJ4 and doesnt have a 'nested test', there shouldn't be any others with the same problem as this test.

        Then we can separately address your solr.solr.home problem within the base classes as you suggest, this is a separate problem I think.

        Show
        Robert Muir added a comment - There are other tests in Solr that have the same problem (solr.solr.home cross-test environment pollution) I don't think this is the same problem? This is just ensuring the class extends SolrTestCaseJ4 and doesnt have a 'nested test', there shouldn't be any others with the same problem as this test. Then we can separately address your solr.solr.home problem within the base classes as you suggest, this is a separate problem I think.
        Hide
        Steve Rowe added a comment -

        There are other tests in Solr that have the same problem (solr.solr.home cross-test environment pollution), and it would be nice if it were possible to just touch SolrTestCaseJ4, rather than each problematic Solr test; I was just asking if you had found some issue that disallowed this approach.

        Show
        Steve Rowe added a comment - There are other tests in Solr that have the same problem ( solr.solr.home cross-test environment pollution), and it would be nice if it were possible to just touch SolrTestCaseJ4, rather than each problematic Solr test; I was just asking if you had found some issue that disallowed this approach.
        Hide
        Robert Muir added a comment -

        Although your patch switches the test so that it extends SolrTestCaseJ4, you didn't make these changes in SolrTestCaseJ4 itself, so that each test doesn't have to individually host these kinds of changes - is there some reason for that?

        I didnt do anything except fix the test-within-a-test part of it...

        previously this class extended TestCase, but inside it was an inner class that extended SolrTestCase [yet didnt have any tests].
        that was the cause of your problem I think, the inner class just extends Object in this patch and the outer extends SolrTestCaseJ4.

        Show
        Robert Muir added a comment - Although your patch switches the test so that it extends SolrTestCaseJ4, you didn't make these changes in SolrTestCaseJ4 itself, so that each test doesn't have to individually host these kinds of changes - is there some reason for that? I didnt do anything except fix the test-within-a-test part of it... previously this class extended TestCase, but inside it was an inner class that extended SolrTestCase [yet didnt have any tests] . that was the cause of your problem I think, the inner class just extends Object in this patch and the outer extends SolrTestCaseJ4.
        Hide
        Steve Rowe added a comment -

        Steven, here's a fix for that test. I think it should resolve the problem in your IDE

        Cool, thanks, I'll test it out tonight.

        Although your patch switches the test so that it extends SolrTestCaseJ4, you didn't make these changes in SolrTestCaseJ4 itself, so that each test doesn't have to individually host these kinds of changes - is there some reason for that?

        Show
        Steve Rowe added a comment - Steven, here's a fix for that test. I think it should resolve the problem in your IDE Cool, thanks, I'll test it out tonight. Although your patch switches the test so that it extends SolrTestCaseJ4, you didn't make these changes in SolrTestCaseJ4 itself, so that each test doesn't have to individually host these kinds of changes - is there some reason for that?
        Hide
        Robert Muir added a comment -

        Steven, here's a fix for that test. I think it should resolve the problem in your IDE

        Show
        Robert Muir added a comment - Steven, here's a fix for that test. I think it should resolve the problem in your IDE
        Hide
        Steve Rowe added a comment -

        Patch for trunk that adds Ant build integration.

        Show
        Steve Rowe added a comment - Patch for trunk that adds Ant build integration.
        Hide
        Steve Rowe added a comment -

        Patch for 3.X branch. Adds in Ant build.xml file integration, so that tasks can be run from IntelliJ.

        This patch does not include lucene/backwards as a module - I'm not sure if it's even possible to achieve the 3.0.2 jar compile-time, lucene 3.X test-time setup with IntelliJ's configuration. When I setup 3.0.2 jar as a compile-time dependency and the lucene module as a lower priority test-time dependency, the tests compiled and ran, but about 20 tests failed.

        Show
        Steve Rowe added a comment - Patch for 3.X branch. Adds in Ant build.xml file integration, so that tasks can be run from IntelliJ. This patch does not include lucene/backwards as a module - I'm not sure if it's even possible to achieve the 3.0.2 jar compile-time, lucene 3.X test-time setup with IntelliJ's configuration. When I setup 3.0.2 jar as a compile-time dependency and the lucene module as a lower priority test-time dependency, the tests compiled and ran, but about 20 tests failed.
        Hide
        Robert Muir added a comment -

        Yeah, I thought of that, but e.g. DIH's TestContentStreamDataSource directly extends TestCase - maybe it wouldn't be a problem to switch, I dunno - I'll look into it.

        I know, but really it should not. I think the test just needs to be fixed separately, i think it extends TestCase because it has one of those Test-within-a-Test things.

        Show
        Robert Muir added a comment - Yeah, I thought of that, but e.g. DIH's TestContentStreamDataSource directly extends TestCase - maybe it wouldn't be a problem to switch, I dunno - I'll look into it. I know, but really it should not. I think the test just needs to be fixed separately, i think it extends TestCase because it has one of those Test-within-a-Test things.
        Hide
        Steve Rowe added a comment -

        sounds like we should re-open LUCENE-2398 and fix these test problems so they work from Intellij?

        Sounds good to me.

        BTW, unlike Ant, IntelliJ doesn't appear to support forking VM process per test class, so that route isn't available.

        Can you supply a patch with the fixes to solr.solr.home in tearDown?

        Will do.

        Another idea would be to add your clear to AbstractSolrTestCase/SolrTestCaseJ4's tearDown/afterClass to prevent any problems?

        Yeah, I thought of that, but e.g. DIH's TestContentStreamDataSource directly extends TestCase - maybe it wouldn't be a problem to switch, I dunno - I'll look into it.

        Show
        Steve Rowe added a comment - sounds like we should re-open LUCENE-2398 and fix these test problems so they work from Intellij? Sounds good to me. BTW, unlike Ant, IntelliJ doesn't appear to support forking VM process per test class, so that route isn't available. Can you supply a patch with the fixes to solr.solr.home in tearDown? Will do. Another idea would be to add your clear to AbstractSolrTestCase/SolrTestCaseJ4's tearDown/afterClass to prevent any problems? Yeah, I thought of that, but e.g. DIH's TestContentStreamDataSource directly extends TestCase - maybe it wouldn't be a problem to switch, I dunno - I'll look into it.
        Hide
        Robert Muir added a comment -

        Thanks Steven. sounds like we should re-open LUCENE-2398 and fix these test problems so they work from Intellij?
        Can you supply a patch with the fixes to solr.solr.home in tearDown?
        Another idea would be to add your clear to AbstractSolrTestCase/SolrTestCaseJ4's tearDown/afterClass to prevent any problems?

        Show
        Robert Muir added a comment - Thanks Steven. sounds like we should re-open LUCENE-2398 and fix these test problems so they work from Intellij? Can you supply a patch with the fixes to solr.solr.home in tearDown? Another idea would be to add your clear to AbstractSolrTestCase/SolrTestCaseJ4's tearDown/afterClass to prevent any problems?
        Hide
        Steve Rowe added a comment -

        Can you provide any information on tests that are giving you trouble?

        There is a class of problems involving "leakage" of solr.solr.home property setting between test classes. AFAICT, Solr's resource loader uses the current directory when that property is not set. Some tests use the current directory, and others set solr.solr.home. When a test that expects to use the current directory follows a test that has set solr.solr.home, things go wrong.

        Adding System.clearProperty("solr.solr.home"); to tearDown() in DIH's TestContentStreamDataSource fixed the problem for me in DIH. Solr has several tests that set solr.solr.home, e.g. TestSolrCoreProperties, and when I added clearProperty() to tearDown() in that class, several following tests expecting default current directory stopped failing.

        There is another class of problems related to network connections, in Solr, that I haven't tracked down yet.

        Show
        Steve Rowe added a comment - Can you provide any information on tests that are giving you trouble? There is a class of problems involving "leakage" of solr.solr.home property setting between test classes. AFAICT, Solr's resource loader uses the current directory when that property is not set. Some tests use the current directory, and others set solr.solr.home . When a test that expects to use the current directory follows a test that has set solr.solr.home , things go wrong. Adding System.clearProperty("solr.solr.home"); to tearDown() in DIH's TestContentStreamDataSource fixed the problem for me in DIH. Solr has several tests that set solr.solr.home , e.g. TestSolrCoreProperties , and when I added clearProperty() to tearDown() in that class, several following tests expecting default current directory stopped failing. There is another class of problems related to network connections, in Solr, that I haven't tracked down yet.
        Hide
        Robert Muir added a comment -

        DIH and Solr unit test runs still don't fully pass for me, but all other modules' test runs pass.

        Can you provide any information on tests that are giving you trouble?
        We could re-open LUCENE-2398 also. really it would be nice if all tests worked from these IDEs.

        Show
        Robert Muir added a comment - DIH and Solr unit test runs still don't fully pass for me, but all other modules' test runs pass. Can you provide any information on tests that are giving you trouble? We could re-open LUCENE-2398 also. really it would be nice if all tests worked from these IDEs.
        Hide
        Erick Erickson added a comment -

        Steven:

        That worked like a champ, all I had to do was set the project-level JDK and then run tests.

        The only other anomaly (and it's not causing me any problems) is that on the project settings page, there are "circular dependencies"...
        1. queries, misc, common, remote
        2. solr, extraction

        FWIW
        Erick

        Show
        Erick Erickson added a comment - Steven: That worked like a champ, all I had to do was set the project-level JDK and then run tests. The only other anomaly (and it's not causing me any problems) is that on the project settings page, there are "circular dependencies"... 1. queries, misc, common, remote 2. solr, extraction FWIW Erick
        Hide
        Steve Rowe added a comment -

        New patch:

        • puts back per-module sdk inheritance in the *.iml files
        • adds a module unit test run for the Solr Clustering contrib, now that it has been re-enabled.
        • other miscellaneous improvements

        DIH and Solr unit test runs still don't fully pass for me, but all other modules' test runs pass.

        Show
        Steve Rowe added a comment - New patch: puts back per-module sdk inheritance in the *.iml files adds a module unit test run for the Solr Clustering contrib, now that it has been re-enabled. other miscellaneous improvements DIH and Solr unit test runs still don't fully pass for me, but all other modules' test runs pass.
        Hide
        Steve Rowe added a comment -

        Hi Erick,

        I have to go in to each module and re-select "project sdk" on the dependencies tab, even though it looks like it's already selected!

        I removed a chunk of configuration from the *.iml files that sets this, I think - I'll post a patch shortly that puts the per-module project SDK inheritance back, and should hopefully address the problem you're seeing.

        Show
        Steve Rowe added a comment - Hi Erick, I have to go in to each module and re-select "project sdk" on the dependencies tab, even though it looks like it's already selected! I removed a chunk of configuration from the *.iml files that sets this, I think - I'll post a patch shortly that puts the per-module project SDK inheritance back, and should hopefully address the problem you're seeing.
        Hide
        Erick Erickson added a comment -

        This is waaaaay cool. I happen to be setting up a new machine so I'm field testing it.

        The only issue I'm seeing is that on a Windows7 machine, I have to go in to each module and re-select "project sdk" on the dependencies tab, even though it looks like it's already selected! About drove me crazy for a bit..

        I doubt there's anything to be done about it in the patch, more of a documentation issue.

        I'll update the bit on "how to contribute" to reference this patch real soon now with a warning about the above unless someone suggests a better way...

        Erick

        Show
        Erick Erickson added a comment - This is waaaaay cool. I happen to be setting up a new machine so I'm field testing it. The only issue I'm seeing is that on a Windows7 machine, I have to go in to each module and re-select "project sdk" on the dependencies tab, even though it looks like it's already selected! About drove me crazy for a bit.. I doubt there's anything to be done about it in the patch, more of a documentation issue. I'll update the bit on "how to contribute" to reference this patch real soon now with a warning about the above unless someone suggests a better way... Erick
        Hide
        Steve Rowe added a comment -

        All compilation succeeds for me with the attached patch under IntelliJ IDEA 9.0.3 on Windows Vista 64-bit with Java6 1.6.0_13 64-bit.

        Except for the Solr and the DIH modules, all modules' JUnit test runs succeed for me.

        The DIH module JUnit test run succeeds if I add code to TestContentStreamDataSource to clear the solr.solr.home property in tearDown(). Without this code, the solr.solr.home property setting introduced in leaks into following JUnit tests and cause the resource loader to look in the wrong place. (The attached patch does not contain the property clearing code, since I wasn't sure that was the right way to go.)

        The Solr module JUnit test run shares the solr.solr.home leakage problem, at a minimum in TestSolrCoreProperties, and likely in other tests too. However, I didn't track down all of those problems because I got a bunch of failures related to refused connections and HTTP 500 errors. These may be because my firewall settings don't allow these connections, but I haven't tracked the problem(s) down yet, so I'm not sure.

        I plan on making another patch with a similar setup for the 3.X branch.

        Show
        Steve Rowe added a comment - All compilation succeeds for me with the attached patch under IntelliJ IDEA 9.0.3 on Windows Vista 64-bit with Java6 1.6.0_13 64-bit. Except for the Solr and the DIH modules, all modules' JUnit test runs succeed for me. The DIH module JUnit test run succeeds if I add code to TestContentStreamDataSource to clear the solr.solr.home property in tearDown() . Without this code, the solr.solr.home property setting introduced in leaks into following JUnit tests and cause the resource loader to look in the wrong place. (The attached patch does not contain the property clearing code, since I wasn't sure that was the right way to go.) The Solr module JUnit test run shares the solr.solr.home leakage problem, at a minimum in TestSolrCoreProperties , and likely in other tests too. However, I didn't track down all of those problems because I got a bunch of failures related to refused connections and HTTP 500 errors. These may be because my firewall settings don't allow these connections, but I haven't tracked the problem(s) down yet, so I'm not sure. I plan on making another patch with a similar setup for the 3.X branch.
        Hide
        Steve Rowe added a comment -

        I have tested compilation and test runs on IntelliJ IDEA 9.0.3 on Windows Vista 64-bit with Java 1.6.0_13 64-bit.

        All compilation succeeds with the attached patch.

        Except for the Solr module and Solr DIH contrib module test runs, each module's test run completes successfully for me.

        The DIH test run works for me if I add code to TestSolrCoreProperties to clear the solr.solr.home property in tearDown() - without this code, the solr.solr.home setting leaks into the following tests and causes the resource loader to look in the wrong places. The attached patch does not include this code, though. When run individually, all DIH tests pass.

        The Solr test run shares the solr.solr.home leakage problem with DIH test run, at least in TestContentStreamDataSource, and likely in other places as well. However, I wasn't able to track down the causes of 500 HTTP errors and connection failures in some tests. Maybe I didn't have my firewall set up for running these tests in IntelliJ, I'm not sure - I didn't track these issues down.

        I plan on creating a similar setup for the 3.X branch.

        Show
        Steve Rowe added a comment - I have tested compilation and test runs on IntelliJ IDEA 9.0.3 on Windows Vista 64-bit with Java 1.6.0_13 64-bit. All compilation succeeds with the attached patch. Except for the Solr module and Solr DIH contrib module test runs, each module's test run completes successfully for me. The DIH test run works for me if I add code to TestSolrCoreProperties to clear the solr.solr.home property in tearDown() - without this code, the solr.solr.home setting leaks into the following tests and causes the resource loader to look in the wrong places. The attached patch does not include this code, though. When run individually, all DIH tests pass. The Solr test run shares the solr.solr.home leakage problem with DIH test run, at least in TestContentStreamDataSource, and likely in other places as well. However, I wasn't able to track down the causes of 500 HTTP errors and connection failures in some tests. Maybe I didn't have my firewall set up for running these tests in IntelliJ, I'm not sure - I didn't track these issues down. I plan on creating a similar setup for the 3.X branch.

          People

          • Assignee:
            Unassigned
            Reporter:
            Steve Rowe
          • Votes:
            3 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development