Patch implementing the idea, along with a script to fix existing patches against the old structure to be against the new structure.

Run this svn move script before applying the patch:

svn mv --parents lucene/src/java lucene/core/src/java
svn mv --parents lucene/src/test lucene/core/src/test
svn mv --parents lucene/src/resources lucene/core/src/resources
svn mv lucene/src/site lucene/site
svn mv --parents lucene/src/test-framework/java lucene/test-framework/src/java
svn mv --parents lucene/src/test-framework/resources lucene/test-framework/src/resources
svn mv --parents lucene/src/tools/java lucene/tools/src/java
svn mv --parents lucene/src/tools/javadoc lucene/tools/javadoc
svn mv --parents lucene/src/tools/prettify lucene/tools/prettify
svn rm lucene/src
svn mv --parents dev-tools/maven/lucene/src/pom.xml.template dev-tools/maven/lucene/core/pom.xml.template
svn mv --parents dev-tools/maven/lucene/src/test-framework/pom.xml.template dev-tools/maven/lucene/test-framework/pom.xml.template
svn rm dev-tools/maven/lucene/src

I think this is ready to go.

Can't say I object to the patch if it has a script to fix old patches, will go back to 3x too, and if it will speed up the build system?

(Note: I didn't dig into LUCENE-3754, but its my impression that it would?)

My only suggestions (more like personal requests actually):

• can 'ant test-core' be an alias for 'test' inside lucene/core? And same with javadocs-core? It might work
  already in the patch (I didnt test), but I noticed this trips me up for solr as well... just used to typing it i guess.
• can we give a heads up to the list first? i know i have a bunch of old checkouts where if i ran 'svn up'
  i'd have to deal with a hellacious merge conflict, but dumping out to a patch, upgrading that patch with the tool,
  and then applying that to a new checkout with patch --merge would probably be easier (I'd only have to deal with
  the "real" conflicts then)

Can't say I object to the patch if it has a script to fix old patches

Yes.

will go back to 3x too

That's my plan, yes.

and if it will speed up the build system? (Note: I didn't dig into LUCENE-3754, but its my impression that it would?)

I don't know. I figured it would be about the same speed, but I'll check to be sure. (I think ant build-contrib in lucene/ is the right thing to use for this.) LUCENE-3754 will speed up the Solr build, since it depends on lucene-core jar, but a lot (all?) of the Lucene build depends on un-jarred build outputs, and so won't be affected, AFAIK.

can 'ant test-core' be an alias for 'test' inside lucene/core? And same with javadocs-core? It might work already in the patch (I didnt test), but I noticed this trips me up for solr as well... just used to typing it i guess.

I'll add them if they don't already work (not sure).

can we give a heads up to the list first? i know i have a bunch of old checkouts where if i ran 'svn up' i'd have to deal with a hellacious merge conflict, but dumping out to a patch, upgrading that patch with the tool, and then applying that to a new checkout with patch --merge would probably be easier (I'd only have to deal with the "real" conflicts then)

Yes, I agree a heads-up to the dev list first would be good - I was planning on giving 24 hours notice before committing.

I ran ant build-contrib from lucene/, both with and without running ant clean first (the timings below exclude ant clean) - these timings are best of 5 runs each:

So, slightly faster for the from-scratch case, and slightly slower from the already-built case - I think this slight slowdown is okay.

I'll run similar tests from modules/ and solr/ and report back.

I ran ant compile-test from modules/ and solr/, both with and without running ant clean from the top level first (the timings below exclude ant clean) - these timings are best of 5 runs each:

These are basically unchanged.

This version of the patch adds:

• alias targets test-core and javadocs-core to lucene/core/build.xml;
• alias target javadocs-core to lucene/test-framework/build.xml; and
• a new target compile-test to lucene/build.xml, which compiles all modules' non-test and test sources (the existing target build-contrib also builds contribs' jars).

Also, I added mkdir ${build.dir} to the javadocs targets in lucene/core/ and lucene/test-framework/, to make sure the target dir for the javadocs jars exists. (This isn't necessary in the current setup because the target dir is lucene/build/, which gets created when ${javadoc.dir} is created by these targets.)

Final version of the patch:

• speeds up ant test from lucene/ by making test-contrib depend on compile-test instead of build-contrib (jars don't need to be built for testing.)
• includes more *.compiled property handling to avoid recompiling lucene-core and lucene-test-framework.

Post-patch ant compile-test times under lucene/ are now 31s after running ant clean, and 7s when already compiled.

modules/ and solr/ timings did not change.

Post-patch ant compile-test times under lucene/ are now 31s after running ant clean, and 7s when already compiled.

Great! Thanks for doing this refactoring!

Steve, are you going to merge running the tests somehow too? I ask because I have a backlog of trying to integrate automatically load-balancing parallel tests; if we had a single test task this would mean one could pattern-scope the tests to be executed and (if not restricted) all of the tests would run in a single balancing scope (Mike pointed out that currently a lot of time is spent on waiting for the last test in each spawned).

Just asking, no obligations to change anything.

Hi Dawid,

Steve, are you going to merge running the tests somehow too?

No, my patch doesn't do that. Sounds really cool though. I think Mike M.'s python test runner does this?

if we had a single test task this would mean one could pattern-scope the tests to be executed and (if not restricted) all of the tests would run in a single balancing scope (Mike pointed out that currently a lot of time is spent on waiting for the last test in each spawned).

Yes, but this doesn't really test any dependencies that different contribs/modules might have right? Or maybe i misunderstand what you are suggesting.

I think Mike's python runner "cheats" and makes a huge classpath... sure its faster, but, it loses test coverage because modules could have
bogus dependencies and we would never know.

Yes, this is what Mike's python magic does – puts a fraction of all tests on each slave, then assigns on-demand. I reimplemented this in Java and it works quite nice as it can even consume precomputed statistics to make better estimates. The problem is as always in details (which I won't trouble you with at the moment). Robert is also right that re-running with different parameters/ JVM settings/ classpath will require multiple runs... for a moment I had a hope to neglect this

The speedups aren't dramatic, especially after Robert worked on the longest tests, so this is not a huge priority. Interestingly, Maven's latest release of surefire also includes an option to do balanced parallel tests (on separate vms). Didn't experiment with it yet.

Maven's latest release of surefire also includes an option to do balanced parallel tests (on separate vms).

I'm not sure that's exactly true? SUREFIRE-799 includes a discussion of some limitations of the new capabilities.

The speedups aren't dramatic, especially after Robert worked on the longest tests, so this is not a huge priority.

Well, from time to time we try to clean up the tests... but i think it only takes a few weeks for us to 'lose' all the benefits.
I'm not trying to say our tests get crappier, a lot of the reasons for the slowdown is really just more test coverage...!

Basically, I don't think we can view speeding up slow tests as one-off tasks... its gotta be a continuous maintenance thing.

For this reason, its important to speed up the other parts of the build (like rebuilding jars over and over again
such as this issue), and also to integrate the nice random test framework work etc you did... these are more 'permanent'
wins.

However, I'm just concerned about adopting mike's cheating, because I like the idea of 'ant test' being a total pain
to catch broken stuff. New modules, etc are added often, the build system is refactored, dependencies upgraded, etc etc,
and I'm concerned we would somehow screw ourselves up and release stuff with bogus dependencies.

@Robert. You're right, dependencies (and their order) may screw up things. I was tinkering with an idea that testing could be a multi-pass process, then it would be possible to collect all the "specs" in one pass and then run everything in another pass... but it temporarily exceeds my time limits to actually implement this

@Steven. That was my impression from reading the release notes and from looking at the code:

Surefire 2.12 now supports parallel forks, which can also be run
in runOrder="balanced", which is a nice feature for long-running
tests.

Ah, I see – I didn't mean "balanced" in the sense of dynamic balancing, the tests are statically assigned, but they do run in separate concurrent slave JVMs and that's still something

I meant to end with a "?" – do you know if my understanding is correct, Steve?

Nah, just read the docs – you're right, this is not multiple-test-per-jvm allocation. I thought it was. Interestingly I did a small surefire replacement for that ant task (with a few incompatibilities) that does run multiple jvms for testing. If you want to check it out and provide feedback, it'd be awesome. An example of use is in this POM:

https://github.com/carrotsearch/randomizedtesting/blob/master/integration-maven/junit4-maven-plugin-tests/src/it/01-basic-test/pom.xml

Interestingly I did a small surefire replacement for that ant task (with a few incompatibilities) that does run multiple jvms for testing. If you want to check it out and provide feedback, it'd be awesome.

Sure, I'll take a look later this week.

+1 to commit this change to the build system! That was really needed!

Committed to trunk.

Working on backport to branch_3x now.

Committed to branch_3x.

Now working on fixing broken target generate-maven-artifacts under lucene/ on trunk.

Now working on fixing broken target generate-maven-artifacts under lucene/ on trunk.

This is now fixed - I committed the fix to branch_3x as well, where for some reason generate-maven-artifacts was already functional without the fix....

I'll leave this issue open for a day or two in case more problems crop up.

Now working on fixing broken target generate-maven-artifacts under lucene/ on trunk.

This is now fixed - I committed the fix to branch_3x as well, where for some reason generate-maven-artifacts was already functional without the fix....

I understand now why generate-maven-artifacts was functional in branch_3x before I committed the backport: as a result of svn move'ing files, the original directories are retained until svn commit is done.

I've seen this enough times now that I should remember: committing to subversion can change build behavior.

The Jenkins builds look stable now.

Any additional problems can be addressed in new issues.

Steven: There is still a problem with the clover reports: The directories for generating the reports don't correctly identify "Tests", "Test Results" and the "Class files". The ant task's "generate-clover-reports" seems to use the wrong paths. It looks like it only catches some tests, most test classes are listed under

We need maybe a more "universal" fileset for <testsources/> and <testresults/>. Maybe use the first one for contrib somehow globally:

<fileset dir="contrib" id="clover.contrib.test.src.files">
 <include name="**/test/**/*.java"/>
</fileset>

(e.g. using dir="."). Same for test-results, this is also wrong:

<fileset dir="${build.dir}" id="clover.test.result.files">
 <include name="**/test/TEST-*.xml" />
 <!-- do not include BW tests -->
 <exclude name="backwards/**"/>
</fileset>

Affects both 3.x and trunk. I don't want to fix it, you know better where al those different build dirs could live.

Ah, here are the reports: https://builds.apache.org/job/Lucene-trunk/clover/ (you can see the bugs when you look on the left frame and switch between the tabs "Classes", "Tests", "Results". Most tests are now listed at "Classes", which is wrong.

Hi Uwe, I'm looking into it.

I committed clover test source pattern changes to trunk and branch_3x, but I haven't been able to access Jenkins since (service unavailable/proxy errors), so I'm not sure if they were effective. I'll keep checking.