Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.4, 4.0-ALPHA
    • Component/s: Build
    • Labels:
      None

      Description

      As discussed some in SOLR-2002 (but that issue is long and hard to follow), I think we should rewrite the solr build system.

      Its slow, cumbersome, and messy, and makes it hard for us to improve things.

      1. SOLR-2452.dir.reshuffle.sh
        2 kB
        Steve Rowe
      2. SOLR-2452-post-reshuffling.patch
        57 kB
        Steve Rowe
      3. SOLR-2452-post-reshuffling.patch
        59 kB
        Steve Rowe
      4. SOLR-2452.dir.reshuffle.sh
        2 kB
        Steve Rowe
      5. SOLR-2452-post-reshuffling.patch
        132 kB
        Steve Rowe
      6. SOLR-2452.diffSource.py.patch.zip
        4.90 MB
        Steve Rowe
      7. SOLR-2452.dir.reshuffle.branch_3x.sh
        5 kB
        Steve Rowe
      8. SOLR-2452.post.dir.reshuffle.branch_3x.patch
        252 kB
        Steve Rowe

        Issue Links

          Activity

          Hide
          Robert Muir added a comment -

          I brought my previous patch up to date and committed it to

          https://svn.apache.org/repos/asf/lucene/dev/branches/solr2452

          i ripped all the existing stuff out of the solr build: we can always add stuff back but I wanted to start lean and mean. compile/test/clean/dependencies/etc should work, and are extended from lucene's build.

          appreciate anyone who wants to spend time trying to help.

          Show
          Robert Muir added a comment - I brought my previous patch up to date and committed it to https://svn.apache.org/repos/asf/lucene/dev/branches/solr2452 i ripped all the existing stuff out of the solr build: we can always add stuff back but I wanted to start lean and mean. compile/test/clean/dependencies/etc should work, and are extended from lucene's build. appreciate anyone who wants to spend time trying to help.
          Hide
          Steve Rowe added a comment -

          Robert, I synch'd this up to trunk yesterday (and it's already out of date!), and made a few minor changes.

          One of the changes I made was switching the solr/contrib/analysis-extras/ layout to the conventional Maven layout, to mirror all the other Solr contribs:

          src/
          +--- main/
          |    +--- java/   
          +--- test/
               +--- java/
               +--- resources/
          

          But I see that the layout for the new modules/suggest/ module you just added (LUCENE-2995) is:

          src/
          +--- java/
          +--- test/
          

          (and if it had had any test resources, I assume they would be under src/test-files/).

          I think there should just be one layout for all Lucene & Solr modules (in the abstract sense of the word).

          I would be fine with switching all the Solr contribs to use the same layout as the new suggest module.

          Thoughts?

          Show
          Steve Rowe added a comment - Robert, I synch'd this up to trunk yesterday (and it's already out of date!), and made a few minor changes. One of the changes I made was switching the solr/contrib/analysis-extras/ layout to the conventional Maven layout, to mirror all the other Solr contribs: src/ +--- main/ | +--- java/ +--- test/ +--- java/ +--- resources/ But I see that the layout for the new modules/suggest/ module you just added ( LUCENE-2995 ) is: src/ +--- java/ +--- test/ (and if it had had any test resources, I assume they would be under src/test-files/ ). I think there should just be one layout for all Lucene & Solr modules (in the abstract sense of the word). I would be fine with switching all the Solr contribs to use the same layout as the new suggest module. Thoughts?
          Hide
          Robert Muir added a comment -

          I think there should just be one layout for all Lucene & Solr modules (in the abstract sense of the word).

          I would be fine with switching all the Solr contribs to use the same layout as the new suggest module.

          +1, and thanks for iterating on this issue more!

          I would like to see this stuff consistent, the particulars of whatever that layout is, do not matter to me.
          My only thoughts are that I think this should be fixed in trunk, and also backported to branch 3.x to make merging easier...

          Show
          Robert Muir added a comment - I think there should just be one layout for all Lucene & Solr modules (in the abstract sense of the word). I would be fine with switching all the Solr contribs to use the same layout as the new suggest module. +1, and thanks for iterating on this issue more! I would like to see this stuff consistent, the particulars of whatever that layout is, do not matter to me. My only thoughts are that I think this should be fixed in trunk, and also backported to branch 3.x to make merging easier...
          Hide
          Steve Rowe added a comment -

          Okay, I'll standardize on the following layout, since it's used by the vast majority of Lucene/Solr modules, and so will wreak the least havoc:

          src/
          +--- java/
          +--- resources/       (optional)
          +--- test/
          +--- test-files/      (optional)
          +--- tools/           (optional)
          
          Show
          Steve Rowe added a comment - Okay, I'll standardize on the following layout, since it's used by the vast majority of Lucene/Solr modules, and so will wreak the least havoc: src/ +--- java/ +--- resources/ (optional) +--- test/ +--- test-files/ (optional) +--- tools/ (optional)
          Hide
          Steve Rowe added a comment -

          Robert, I'm merging trunk to the SOLR-2452 branch, and I noticed that SOLR-2279 introduced a solr.directoryFactory <sysproperty> element in the <junit> task invocation for all the Solr contribs.

          I'm trying to figure out how to support this in the consolidated build without duplicating code. My current thinking is to introduce an optional <element> in the "test-macro" definition to supply the <sysproperty> to the <junit> task, and then make new <junit-sequential-solr> and <junit-parallel-solr> targets, including the solr.directoryFactory <sysprop> element in each <test-macro> invocation. Then a new common-solr.test target would depend on the junit-*-solr targets.

          Thoughts?

          Show
          Steve Rowe added a comment - Robert, I'm merging trunk to the SOLR-2452 branch, and I noticed that SOLR-2279 introduced a solr.directoryFactory <sysproperty> element in the <junit> task invocation for all the Solr contribs. I'm trying to figure out how to support this in the consolidated build without duplicating code. My current thinking is to introduce an optional <element> in the "test-macro" definition to supply the <sysproperty> to the <junit> task, and then make new <junit-sequential-solr> and <junit-parallel-solr> targets, including the solr.directoryFactory <sysprop> element in each <test-macro> invocation. Then a new common-solr.test target would depend on the junit-*-solr targets. Thoughts?
          Hide
          Robert Muir added a comment -

          Steven, I think there we should just set the property in lucene/common-build.xml, in the reusable 'test' target. I've already done this with a couple others if you look

          Show
          Robert Muir added a comment - Steven, I think there we should just set the property in lucene/common-build.xml, in the reusable 'test' target. I've already done this with a couple others if you look
          Hide
          Robert Muir added a comment -

          Oh, and, by the way thanks for picking up on that branch... hopefully I can come in and help soon, but its really a relief you are working on it.

          Show
          Robert Muir added a comment - Oh, and, by the way thanks for picking up on that branch... hopefully I can come in and help soon, but its really a relief you are working on it.
          Hide
          Steve Rowe added a comment -

          Steven, I think there we should just set the property in lucene/common-build.xml, in the reusable 'test' target. I've already done this with a couple others if you look

          Thanks, I added it to the junit task invocation in the test-macro definition.

          Show
          Steve Rowe added a comment - Steven, I think there we should just set the property in lucene/common-build.xml, in the reusable 'test' target. I've already done this with a couple others if you look Thanks, I added it to the junit task invocation in the test-macro definition.
          Hide
          Robert Muir added a comment -

          Bulk move 3.2 -> 3.3

          Show
          Robert Muir added a comment - Bulk move 3.2 -> 3.3
          Hide
          Steve Rowe added a comment -

          Forgot to include the issue number in the comment, so it's not showing up here, but I just committed a merge with trunk up to r1135758. Here's the ViewVC link: http://svn.apache.org/viewvc?view=revision&revision=1135759

          Show
          Steve Rowe added a comment - Forgot to include the issue number in the comment, so it's not showing up here, but I just committed a merge with trunk up to r1135758. Here's the ViewVC link: http://svn.apache.org/viewvc?view=revision&revision=1135759
          Hide
          Steve Rowe added a comment -

          This Bash script and patch together eliminate solr/src/ and move everything under it up one level, except for solr/src/java/, solr/src/test/ and solr/src/test-files/, which are moved under a new directory solr/core/:

          solr/
          +---core/
          +---dev-tools/
          +---scripts/
          +---site-src/
          +---solrj/
          +---test-framework/
          +---webapp/
              +---web/
          ...
          

          Additionally:

          1. I merged solr/src/common/ into the solrj/ directory, and moved all o.a.s.client.solrj and o.a.s.common tests from the core tests to solrj/src/test/; and
          2. I merged src/webapp/src/ into the core/src/java/ directory.

          The philosophy at work here is to keep all sources used to build an artifact under a single directory.

          Compilation and testing works, though the core<->solrj dependencies are unusual: non-test core depends on non-test solrj, but solrj tests depend on core tests. (The IntelliJ and Maven builds need fudging to make this work, since their dependency mechanisms are not fine-grained enough to directly support this setup - these changes are also included in the patch.)

          I haven't committed this to the branch because it's fairly radical, and I'd like to make sure nobody objects, but also because tracking trunk changes will become much harder after this change is committed to the branch.

          Robert, would you please review?

          Show
          Steve Rowe added a comment - This Bash script and patch together eliminate solr/src/ and move everything under it up one level, except for solr/src/java/ , solr/src/test/ and solr/src/test-files/ , which are moved under a new directory solr/core/ : solr/ +---core/ +---dev-tools/ +---scripts/ +---site-src/ +---solrj/ +---test-framework/ +---webapp/ +---web/ ... Additionally: I merged solr/src/common/ into the solrj/ directory, and moved all o.a.s.client.solrj and o.a.s.common tests from the core tests to solrj/src/test/ ; and I merged src/webapp/src/ into the core/src/java/ directory. The philosophy at work here is to keep all sources used to build an artifact under a single directory. Compilation and testing works, though the core<->solrj dependencies are unusual: non-test core depends on non-test solrj, but solrj tests depend on core tests. (The IntelliJ and Maven builds need fudging to make this work, since their dependency mechanisms are not fine-grained enough to directly support this setup - these changes are also included in the patch.) I haven't committed this to the branch because it's fairly radical, and I'd like to make sure nobody objects, but also because tracking trunk changes will become much harder after this change is committed to the branch. Robert, would you please review?
          Hide
          Chris Male added a comment -

          I haven't reviewed the changes but just a comment on the philosophy:

          I merged solr/src/common/ into the solrj/ directory

          I don't hugely like this decision. Core Solr code (like SolrDocument for example) should be in Solr core, not in SolrJ which is just a client library. Like Lucene, Solr core should contain all the code it needs. Any other modules (which SolrJ can be considered as one) should then depend on the core.

          Whats the reason for making the choice to put the common code in SolrJ?

          Show
          Chris Male added a comment - I haven't reviewed the changes but just a comment on the philosophy: I merged solr/src/common/ into the solrj/ directory I don't hugely like this decision. Core Solr code (like SolrDocument for example) should be in Solr core, not in SolrJ which is just a client library. Like Lucene, Solr core should contain all the code it needs. Any other modules (which SolrJ can be considered as one) should then depend on the core. Whats the reason for making the choice to put the common code in SolrJ?
          Hide
          Steve Rowe added a comment -

          Whats the reason for making the choice to put the common code in SolrJ?

          The current packaging arrangement has the the common/ code in the solrj jar; similarly, the solr-core jar contains the webapp/src/ code.

          The only thing I've done here is eliminated separate source directories. I haven't changed the packaging.

          Show
          Steve Rowe added a comment - Whats the reason for making the choice to put the common code in SolrJ? The current packaging arrangement has the the common/ code in the solrj jar; similarly, the solr-core jar contains the webapp/src/ code. The only thing I've done here is eliminated separate source directories. I haven't changed the packaging.
          Hide
          Chris Male added a comment -

          I see. Can we address the packaging or is that out of scope of this work?

          Show
          Chris Male added a comment - I see. Can we address the packaging or is that out of scope of this work?
          Hide
          Steve Rowe added a comment -

          SolrJ [...] is just a client library

          That's not all it is; on 8/18/2010 on #lucene IRC, yonik wrote:

          solrj used to not be included in the war, but solr core uses solrj for distributed search

          Show
          Steve Rowe added a comment - SolrJ [...] is just a client library That's not all it is; on 8/18/2010 on #lucene IRC, yonik wrote: solrj used to not be included in the war, but solr core uses solrj for distributed search
          Hide
          Steve Rowe added a comment -

          Can we address the packaging or is that out of scope of this work?

          What did you have in mind?

          Show
          Steve Rowe added a comment - Can we address the packaging or is that out of scope of this work? What did you have in mind?
          Hide
          Chris Male added a comment -

          Hmmm I hadn't considered the issue of SolrJ being used for distributed search.

          Show
          Chris Male added a comment - Hmmm I hadn't considered the issue of SolrJ being used for distributed search.
          Hide
          Chris Male added a comment -

          I think my comments can be addressed later on maybe and shouldn't stop these improvements from going forward so +1

          Show
          Chris Male added a comment - I think my comments can be addressed later on maybe and shouldn't stop these improvements from going forward so +1
          Hide
          Robert Muir added a comment -

          but solrj tests depend on core tests

          Curious why this is? some base classes that could be moved into test-framework instead?

          Show
          Robert Muir added a comment - but solrj tests depend on core tests Curious why this is? some base classes that could be moved into test-framework instead?
          Hide
          Steve Rowe added a comment -

          but solrj tests depend on core tests

          Curious why this is? some base classes that could be moved into test-framework instead?

          At a minimum, o.a.s.client.solrj.SolrJettyTestBase (likely should be moved to another package, given that Solr core o.a.s.servlet.*CacheHeaderTest* tests extend this class) and o.a.s.util.ExternalPaths.

          Show
          Steve Rowe added a comment - but solrj tests depend on core tests Curious why this is? some base classes that could be moved into test-framework instead? At a minimum, o.a.s.client.solrj.SolrJettyTestBase (likely should be moved to another package, given that Solr core o.a.s.servlet.*CacheHeaderTest* tests extend this class) and o.a.s.util.ExternalPaths .
          Hide
          Robert Muir added a comment -

          ok, i was just curious, sounds like something that could possibly be dealt with later.

          I think i said it before too, I find it confusing the way these directories all depend upon each other today and how each one is not its own 'subproject' of the build (that basically acts like a contrib or module itself and states its dependencies). So I would really like to see this fixed.

          However, I think I would recommend thinking about when you want to make the change: it will make merging code up to this branch nearly impossible... is it holding back other changes or is this a final step?

          Show
          Robert Muir added a comment - ok, i was just curious, sounds like something that could possibly be dealt with later. I think i said it before too, I find it confusing the way these directories all depend upon each other today and how each one is not its own 'subproject' of the build (that basically acts like a contrib or module itself and states its dependencies). So I would really like to see this fixed. However, I think I would recommend thinking about when you want to make the change: it will make merging code up to this branch nearly impossible... is it holding back other changes or is this a final step?
          Hide
          Robert Muir added a comment -

          by the way, obviously since you have been doing all the work here, i don't want you to read this as me questioning/objecting to the change, just trying to maybe help save you some sanity... if you don't mind dealing with the merging I would just say go for it.

          Show
          Robert Muir added a comment - by the way, obviously since you have been doing all the work here, i don't want you to read this as me questioning/objecting to the change, just trying to maybe help save you some sanity... if you don't mind dealing with the merging I would just say go for it.
          Hide
          Steve Rowe added a comment -

          However, I think I would recommend thinking about when you want to make the change: it will make merging code up to this branch nearly impossible... is it holding back other changes or is this a final step?

          It's not a final step. All of the targets you removed need to be put back (I counted 40 or so). But I think this will be a minor amount of work comparitively.

          I think for the moment I'll keep iterating on the patch, rather than committing it to the branch, to minimize merge costs, until I have all of the Solr targets re-implemented. I don't think it'll take too long, maybe another day or two.

          Once that's done, I'll commit the moves/copies from the shell script and the patch, then generate a full patch for review. Assuming there are no objections then, I plan to commit within a day or so to minimize merge costs.

          So I hope to have this issue resolved this week.

          Show
          Steve Rowe added a comment - However, I think I would recommend thinking about when you want to make the change: it will make merging code up to this branch nearly impossible... is it holding back other changes or is this a final step? It's not a final step. All of the targets you removed need to be put back (I counted 40 or so). But I think this will be a minor amount of work comparitively. I think for the moment I'll keep iterating on the patch, rather than committing it to the branch, to minimize merge costs, until I have all of the Solr targets re-implemented. I don't think it'll take too long, maybe another day or two. Once that's done, I'll commit the moves/copies from the shell script and the patch, then generate a full patch for review. Assuming there are no objections then, I plan to commit within a day or so to minimize merge costs. So I hope to have this issue resolved this week.
          Hide
          Robert Muir added a comment -

          So I hope to have this issue resolved this week.

          Really? thats awesome!

          Worst case, some of those top-level targets could be literally 'put back' probably with minimal modifications.
          My idea of temporary nuking was to try to start over, extending lucene's build system, as otherwise i got lost in all the xml.

          Show
          Robert Muir added a comment - So I hope to have this issue resolved this week. Really? thats awesome! Worst case, some of those top-level targets could be literally 'put back' probably with minimal modifications. My idea of temporary nuking was to try to start over, extending lucene's build system, as otherwise i got lost in all the xml.
          Hide
          Steve Rowe added a comment -

          This version of the shell script & patch removes Solrj's dependence on Solr core tests, by moving SolrJettyTestBase and ExternalPaths from Solr core to Solr's test-framework – it turns out that these were the only two Solr core test classes that Solrj depended on.

          Show
          Steve Rowe added a comment - This version of the shell script & patch removes Solrj's dependence on Solr core tests, by moving SolrJettyTestBase and ExternalPaths from Solr core to Solr's test-framework – it turns out that these were the only two Solr core test classes that Solrj depended on.
          Hide
          Steve Rowe added a comment -

          This patch restores all of Solr's build targets from trunk; the build system rewrite is feature-complete at this point. (The reshuffling scripts requires no further changes.)

          I moved lucene-lib/ directories to under build/, and eliminated per-contrib clean target actions - instead, ant clean just deletes solr/build/, solr/dist/, solr/package/, and solr/example/solr/lib/.

          Before I commit this patch to the branch, I want to put the build through its paces and examine differences between the outputs from trunk and from branches/solr2452 with this patch applied.

          One difference I found so far: on trunk the Solr create-package target includes duplicate javadocs for some non-contrib modules (core & solrj I think): in the uber-javadocs, and again for the javadocs produced for maven. The per-contrib javadocs, by contrast, are excluded. This makes the compressed binary package about 1.8MB larger than it needs to be, IIRC.

          Show
          Steve Rowe added a comment - This patch restores all of Solr's build targets from trunk; the build system rewrite is feature-complete at this point. (The reshuffling scripts requires no further changes.) I moved lucene-lib/ directories to under build/ , and eliminated per-contrib clean target actions - instead, ant clean just deletes solr/build/ , solr/dist/ , solr/package/ , and solr/example/solr/lib/ . Before I commit this patch to the branch, I want to put the build through its paces and examine differences between the outputs from trunk and from branches/solr2452 with this patch applied. One difference I found so far: on trunk the Solr create-package target includes duplicate javadocs for some non-contrib modules (core & solrj I think): in the uber-javadocs, and again for the javadocs produced for maven. The per-contrib javadocs, by contrast, are excluded. This makes the compressed binary package about 1.8MB larger than it needs to be, IIRC.
          Hide
          Steve Rowe added a comment -

          The solr2452 branch is now up-to-date with trunk, and I've committed to the branch the work that I was keeping as a script/patch pair.

          I think this is ready to commit to trunk.

          For review purposes, I'm attaching the zipped output from python -u diffSource.py trunk branches/solr2452, but the patch is huge, so I don't know how useful it will be. (I had to compress it because it exceeds JIRA's 10MB threshold.)

          I plan on merging the solr2452 branch back to trunk in about 24 hours, and then work on backporting the changes to branch_3x.

          Show
          Steve Rowe added a comment - The solr2452 branch is now up-to-date with trunk, and I've committed to the branch the work that I was keeping as a script/patch pair. I think this is ready to commit to trunk. For review purposes, I'm attaching the zipped output from python -u diffSource.py trunk branches/solr2452 , but the patch is huge, so I don't know how useful it will be. (I had to compress it because it exceeds JIRA's 10MB threshold.) I plan on merging the solr2452 branch back to trunk in about 24 hours, and then work on backporting the changes to branch_3x.
          Hide
          Robert Muir added a comment -

          playing around with the branch, the whole situation looks so much better to me.

          in my opinion we can then go and make other little improvements, make things faster, add new targets, in separate issues... so I think you should just commit before the patch goes out of date.

          maybe we even encounter some serious grief, but I think we should just work thru this in svn.

          great work!

          Show
          Robert Muir added a comment - playing around with the branch, the whole situation looks so much better to me. in my opinion we can then go and make other little improvements, make things faster, add new targets, in separate issues... so I think you should just commit before the patch goes out of date. maybe we even encounter some serious grief, but I think we should just work thru this in svn. great work!
          Hide
          Steve Rowe added a comment -

          Merged with trunk, committed in r1144510. (Forgot to include issue number in log comment.)

          Show
          Steve Rowe added a comment - Merged with trunk, committed in r1144510. (Forgot to include issue number in log comment.)
          Hide
          Steve Rowe added a comment -

          Committed to trunk in r1144761.

          Next up: backport to branch_3x.

          Show
          Steve Rowe added a comment - Committed to trunk in r1144761. Next up: backport to branch_3x.
          Hide
          Erick Erickson added a comment -

          What's the right thing to do here in terms of a patch against the old file structure? Is it reasonable to check out fresh code, hack the patch file to reflect the new paths and apply it to the new structure or must I re-edit the source?

          And is SVN merge smart enough to deal when merging from trunk to 3x when 3x hasn't been changed, or is it better to just wait on it all until the back-port is done?

          The patch I'm trying to apply is 2535, it's pretty small so manually reproducing it isn't a problem.

          I get a little nervous in situations like this <G>....

          Show
          Erick Erickson added a comment - What's the right thing to do here in terms of a patch against the old file structure? Is it reasonable to check out fresh code, hack the patch file to reflect the new paths and apply it to the new structure or must I re-edit the source? And is SVN merge smart enough to deal when merging from trunk to 3x when 3x hasn't been changed, or is it better to just wait on it all until the back-port is done? The patch I'm trying to apply is 2535, it's pretty small so manually reproducing it isn't a problem. I get a little nervous in situations like this <G>....
          Hide
          Steve Rowe added a comment -

          What's the right thing to do here in terms of a patch against the old file structure? Is it reasonable to check out fresh code, hack the patch file to reflect the new paths and apply it to the new structure or must I re-edit the source?

          I'm working on a script to automatically hack patch files against the old file structure. When it's ready, I'll attach it to this issue.

          And is SVN merge smart enough to deal when merging from trunk to 3x when 3x hasn't been changed, or is it better to just wait on it all until the back-port is done?

          AFAIK, SVN merge is not smart enough. I should be able to finish the backport in a day or two, though, so waiting shouldn't be too bad.

          You don't need to wait for these things, and for small patches it may be feasible to manually manage the patch hacking and the 3.x merging. But for larger patches, the effort should be much smaller when the patch hacking script is available and the 3.x backport is finished.

          Show
          Steve Rowe added a comment - What's the right thing to do here in terms of a patch against the old file structure? Is it reasonable to check out fresh code, hack the patch file to reflect the new paths and apply it to the new structure or must I re-edit the source? I'm working on a script to automatically hack patch files against the old file structure. When it's ready, I'll attach it to this issue. And is SVN merge smart enough to deal when merging from trunk to 3x when 3x hasn't been changed, or is it better to just wait on it all until the back-port is done? AFAIK, SVN merge is not smart enough. I should be able to finish the backport in a day or two, though, so waiting shouldn't be too bad. You don't need to wait for these things, and for small patches it may be feasible to manually manage the patch hacking and the 3.x merging. But for larger patches, the effort should be much smaller when the patch hacking script is available and the 3.x backport is finished.
          Hide
          Yonik Seeley added a comment -

          What's the right thing to do here in terms of a patch against the old file structure? Is it reasonable to check out fresh code, hack the patch file to reflect the new paths and apply it to the new structure or must I re-edit the source?

          That's what I did.

          And is SVN merge smart enough to deal when merging from trunk to 3x when 3x hasn't been changed, or is it better to just wait on it all until the back-port is done?

          Apply the changes in 3x however you can (i.e. patch, etc) and then use "svn merge --record-only". http://wiki.apache.org/lucene-java/SvnMerge

          Show
          Yonik Seeley added a comment - What's the right thing to do here in terms of a patch against the old file structure? Is it reasonable to check out fresh code, hack the patch file to reflect the new paths and apply it to the new structure or must I re-edit the source? That's what I did. And is SVN merge smart enough to deal when merging from trunk to 3x when 3x hasn't been changed, or is it better to just wait on it all until the back-port is done? Apply the changes in 3x however you can (i.e. patch, etc) and then use "svn merge --record-only". http://wiki.apache.org/lucene-java/SvnMerge
          Hide
          Steve Rowe added a comment -

          This script, given a patch created with 'svn diff' against trunk pre-SOLR-2452, will write a new patch on the standard output stream with paths fixed up to reflect the post-SOLR2452 structure.

          Usage: perl SOLR-2452.patch.hack.pl < old.patch > new.patch

          I've tested it on a couple of patches I had lying around, and it seems to work.

          Yonik, could you test it on the original patch you said you manually hacked?

          Show
          Steve Rowe added a comment - This script, given a patch created with 'svn diff' against trunk pre- SOLR-2452 , will write a new patch on the standard output stream with paths fixed up to reflect the post-SOLR2452 structure. Usage: perl SOLR-2452 .patch.hack.pl < old.patch > new.patch I've tested it on a couple of patches I had lying around, and it seems to work. Yonik, could you test it on the original patch you said you manually hacked?
          Hide
          Yonik Seeley added a comment -

          The script produced output like this:

          Index: solr/core/src/java/org/apache/solr/core/SolrCore.java
          ===================================================================
          --- solr/src/java/org/apache/solr/core/SolrCore.java    (revision 888880231429dc9c7680375a0a21b1886e59b194)
          +++ solr/src/java/org/apache/solr/core/SolrCore.java    (revision )
          

          Notice that "core" wasn't substituted on the lines starting with — and +++

          Trying to use the resulting patch file, I got:

          /opt/code/lusolr$ patch -p0 < tt.patch
          can't find file to patch at input line 5
          Perhaps you used the wrong -p or --strip option?
          The text leading up to this was:
          --------------------------
          |Index: solr/core/src/java/org/apache/solr/core/SolrCore.java
          |===================================================================
          |--- solr/src/java/org/apache/solr/core/SolrCore.java	(revision 888880231429dc9c7680375a0a21b1886e59b194)
          |+++ solr/src/java/org/apache/solr/core/SolrCore.java	(revision )
          --------------------------
          
          Show
          Yonik Seeley added a comment - The script produced output like this: Index: solr/core/src/java/org/apache/solr/core/SolrCore.java =================================================================== --- solr/src/java/org/apache/solr/core/SolrCore.java (revision 888880231429dc9c7680375a0a21b1886e59b194) +++ solr/src/java/org/apache/solr/core/SolrCore.java (revision ) Notice that "core" wasn't substituted on the lines starting with — and +++ Trying to use the resulting patch file, I got: /opt/code/lusolr$ patch -p0 < tt.patch can't find file to patch at input line 5 Perhaps you used the wrong -p or --strip option? The text leading up to this was: -------------------------- |Index: solr/core/src/java/org/apache/solr/core/SolrCore.java |=================================================================== |--- solr/src/java/org/apache/solr/core/SolrCore.java (revision 888880231429dc9c7680375a0a21b1886e59b194) |+++ solr/src/java/org/apache/solr/core/SolrCore.java (revision ) --------------------------
          Hide
          Steve Rowe added a comment -

          Thanks Yonik - I'll fix it

          Show
          Steve Rowe added a comment - Thanks Yonik - I'll fix it
          Hide
          Steve Rowe added a comment -

          This version of the patch hacking script is fixed so that all paths are modified instead of just the ones on 'Index:' lines

          Show
          Steve Rowe added a comment - This version of the patch hacking script is fixed so that all paths are modified instead of just the ones on 'Index:' lines
          Hide
          Steve Rowe added a comment -

          If there are no objections, I plan on committing the patch hacking script to dev-tools/scripts/ later today.

          Show
          Steve Rowe added a comment - If there are no objections, I plan on committing the patch hacking script to dev-tools/scripts/ later today.
          Hide
          Steve Rowe added a comment -

          I removed the patch hack script attachments, since the version in svn is now better, and I don't want to keep posting fixes here.

          Show
          Steve Rowe added a comment - I removed the patch hack script attachments, since the version in svn is now better, and I don't want to keep posting fixes here.
          Hide
          Steve Rowe added a comment -

          For the record, the svn movement script and post-movement patch against branch_3x.

          Committing shortly.

          Show
          Steve Rowe added a comment - For the record, the svn movement script and post-movement patch against branch_3x. Committing shortly.
          Hide
          Steve Rowe added a comment -

          Committed to branch_3x in r1146191.

          Show
          Steve Rowe added a comment - Committed to branch_3x in r1146191.
          Hide
          Yonik Seeley added a comment -

          On trunk in solr, "ant example" after touching a single source file was 31 seconds before this patch and is now 51. Anyone have any ideas where we lost? (the original description did cite speed as one reason for a rewrite

          Show
          Yonik Seeley added a comment - On trunk in solr, "ant example" after touching a single source file was 31 seconds before this patch and is now 51. Anyone have any ideas where we lost? (the original description did cite speed as one reason for a rewrite
          Hide
          Steve Rowe added a comment - - edited

          On trunk in solr, "ant example" after touching a single source file was 31 seconds before this patch and is now 51. Anyone have any ideas where we lost? (the original description did cite speed as one reason for a rewrite

          I've opened an issue to speed up the build: SOLR-2652

          Show
          Steve Rowe added a comment - - edited On trunk in solr, "ant example" after touching a single source file was 31 seconds before this patch and is now 51. Anyone have any ideas where we lost? (the original description did cite speed as one reason for a rewrite I've opened an issue to speed up the build: SOLR-2652
          Hide
          Steve Rowe added a comment -

          Resolving this issue as trunk and branch_3x seem to have stabilized; when problems related to the Solr build restructuring arise, I plan on opening new issues.

          Show
          Steve Rowe added a comment - Resolving this issue as trunk and branch_3x seem to have stabilized; when problems related to the Solr build restructuring arise, I plan on opening new issues.
          Hide
          Robert Muir added a comment -

          bulk close for 3.4

          Show
          Robert Muir added a comment - bulk close for 3.4

            People

            • Assignee:
              Steve Rowe
              Reporter:
              Robert Muir
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development