Impl. note:

• add an enum flag ifNoTests="warn|fail|ignore".
• add a return property(ies) which could hold the number of executed/ ignored/ failed/ erroneous tests (separate from errorproperty and failureproperty which are already covered in standard junit target); this would allow regular ant fail and logical operators combination.

A patch that adds tests.ifNoTests property. Override your defaults to:

tests.ifNoTests=fail

and then no-test runs will fail as in:

cd lucene
ant test-core  -Dtestcase=TestPayloadsOnVector -Dtests.ifNoTests=fail

results in:

...
test:
   [junit4] <JUnit4> says ??????! Master seed: 2BD9E59CF4296F85
   [junit4] Your default console's encoding may not display certain unicode glyphs: windows-1252
   [junit4] Tests summary: 0 suites, 0 tests

BUILD FAILED
c:\Work\lucene\trunk\lucene\build.xml:49: The following error occurred while executing this line:
c:\Work\lucene\trunk\lucene\common-build.xml:1262: The following error occurred while executing this line:
c:\Work\lucene\trunk\lucene\common-build.xml:905: There were no executed tests: 0 suites, 0 tests

Would it be possible to add a new ant target ("ant run-one-test" or something) that turns this on by default?

Otherwise I imagine nobody would remember to specify this / want to do all the extra typing, to run one test ...

But it can run more than one test? It's more like at-least-one-test...

But it can run more than one test?

True, but I suspect that very few people actually want that? I.e., I think it's OK if the name implies that it will run one test, even if it does the odd globbing under the hood that could result in more than one test? That behavior is more of a corner case ...

Accidentally running more than one test is not nearly as bad as saying BUILD SUCCESSFUL when no test actually ran, I think.

Maybe "ant one-test"? Any other ideas?

ant test-1-class

in short, with Roman numerals: 'testicls'? (don't get any wrong ideas, I'm not a native speaker...

I think I have a better idea. Stay tuned.

As a purely intellectual exercise I decided to investigate whether it's possible to have a "top-level", after-all-the-submodules check for the number of executed tests. Ant really isn't suited for multi-module, hierarchical project layouts; it'd be so much easier with gradle...

Anyway, the attached patch seems to work. It's terribly hacky and terribly ugly, but it does work. Try it from module-level or top-level (lucene or solr, I didn't try to make it work at top-top level).

cd lucene
ant test -Dtests.class=*TestSpellChecker*
...
BUILD SUCCESSFUL

but:

ant test -Dtests.class=*foo*
...
BUILD FAILED
C:\Work\lucene-solr-svn\trunk\lucene\common-build.xml:1278: Not even a single test was executed (a typo in the filter pattern maybe)?

Let me know what you think. Should I commit it? In spite of how ugly it is?

Wow, +1 to commit

I wish we used Python as our build tool!

I think you would like gradle – it's essentially a fully scripted build system (in groovy) but very well thought over.

+1, I've been bitten by this before. This looks great.

No it doesn't, it looks ugly, ugly... But I don't see how it could be done in any other way (under how we use ant and subant calls).

Commit 1539106 from Dawid Weiss in branch 'dev/trunk'
[ https://svn.apache.org/r1539106 ]

LUCENE-5283: Fail the build if ant test didn't execute any tests (everything filtered out).

Committed to trunk. Will let it settle a bit before backporting to 4x.

Hi Dawid,

I was out of offfice, so had no time to review your patch. The current commit seems to leave the checkout unclean after running tests; my response to the failure mail:

it looks like Dawid's commit placed these .tests.totals files in the wrong directory. Should be inside build! Maybe some property is incorrectly initialized.

In addition I will add another comment to the corresponding issue, because of some dependency of the commit to javascript, that’s not fully guaranteed to be available in every JVM (e.g., on FreeBSD we don't have it).

The second thing is a problem on some platforms. We discussed about that already in the past (you know I was fighting pro Javascript), but other convinced me: Javascript is not guaranteed to be a available on every JDK (the JDK only defines the abstract script interface and mandates that some example script engine is avilable, but not which one). So I changed all scripts in out ANT build to use groovy. For Groovy, ideally use the <groovy/> ant task (installed by common-build when you depend on the corresponding install-groovy task. You can also use it with <scriptcondition/> or <script/> but this leads to permgen problems when called in every module. The reason for this is: <script/> creates a new classloader every time and loads a new groovy, while installing the <groovy/> ant task can be reused in sub-modules (so we only need to install top-level).

So I would rewrite the <scriptcondition> to a simple <groovy/> executed before the condition task, which sets a property thats used by the condition. Or alternatively directly throw a BuildException in the groovy without using a scriptcondition at all.

I can provide a patch tomorrow.

Eh... Yeah, thanks Uwe.

Regarding the temporary file – this should have been removed at the end of the build, don't know why it wasn't. I initially placed it under build/ but this wasn't initialized for Solr (if I recall correctly).

I also didn't want to make this any heavier than needed (by loading Groovy, etc) because this gets executed pretty often. I will revert for now and try to work on this in the background.

I'm working on this, don't waste your time, Uwe. Will provide a patch later.

• Rewritten in groovy (with dependency on resolve-groovy).
• Moved the tmp. file to build/

Seems to work. I'm running a full build now.

Commit 1539457 from Dawid Weiss in branch 'dev/trunk'
[ https://svn.apache.org/r1539457 ]

LUCENE-5283: Fail the build if ant test didn't execute any tests (everything filtered out). Take 2: after minor updates and comments from Uwe.

Hi Dawid:
This is obsolete with the groovy code, right?:

<local name="tests.totals.content" />
<loadfile srcFile="${tests.totals.tmpfile}" encoding="UTF-8" property="tests.totals.content" quiet="true" />

One additional thing: On the top-level "test", we should add the dependency to load groovy, to prevent permgen problems (the comment before explains this for the clover one where the same applies). By this, groovy is resolved and installed before any modules are run.

The problem is: Dependencies of targets are executed in any case, also if/unless wuld not run the target itsself. So every submodule would load groovy and then not using it. So it should be loaded before.

Here the fix. Now its much less verbose, because groovy is only loaded at the beginning and not in each module.

It also removes the loadfile.

I actually deferred it on purpose but I see your point in trying to load everything possible up front so that permgen can explode early. The remaining stuff is indeed leftover garbage. Will apply your fix shortly.

Commit 1539475 from Dawid Weiss in branch 'dev/trunk'
[ https://svn.apache.org/r1539475 ]

LUCENE-5283: Fixup to prevent permgens, removed leftover junk (thx Uwe).

Thanks Dawid & Uwe!

Commit 1539975 from Dawid Weiss in branch 'dev/branches/branch_4x'
[ https://svn.apache.org/r1539975 ]

LUCENE-5283: Fail the build if ant test didn't execute any tests (everything filtered out).

Commit 1541116 from Steve Rowe in branch 'dev/branches/branch_4x'
[ https://svn.apache.org/r1541116 ]

LUCENE-5283: Maven config

Commit 1541117 from Steve Rowe in branch 'dev/branches/lucene_solr_4_6'
[ https://svn.apache.org/r1541117 ]

LUCENE-5283: Maven config (merged branch_4x r1541116)

Something seems to have gone wonky with running a single test from the top level...

hossman@frisbee:~/lucene/4x_dev$ ant clean ; ant -Dtestcase=AddBlockUpdateTest test ; find -name TEST-\*.xml
...
BUILD FAILED
/home/hossman/lucene/4x_dev/build.xml:39: The following error occurred while executing this line:
/home/hossman/lucene/4x_dev/lucene/common-build.xml:1281: Not even a single test was executed (a typo in the filter pattern maybe)?

Total time: 46 seconds
hossman@frisbee:~/lucene/4x_dev$

Compare that with...

hossman@frisbee:~/lucene/4x_dev$ cd solr ; ant clean ; ant -Dtestcase=AddBlockUpdateTest test ; find -name TEST-\*.xml
...
BUILD SUCCESSFUL
Total time: 53 seconds
./build/solr-core/test/TEST-org.apache.solr.update.AddBlockUpdateTest.xml
hossman@frisbee:~/lucene/4x_dev/solr$

This was never meant to work at the top level... So it's not a bug, it's an assumption. I'll see if this can be easily fixed – I don't think so because top-level scripts don't have all the logic to load groovy, etc.

Commit 1542198 from Dawid Weiss in branch 'dev/trunk'
[ https://svn.apache.org/r1542198 ]

LUCENE-5283: allow top-level testcase filters.

I've committed a relatively small patch that should allow top-level testcase filtering. The patch is small, but the consequences aren't – it imports common* and extra-targets at the top level so I can't predict all the consequences, really. Works on my machine...

I still have mixed feelings about this issue – it's adds a layer of complexity that doesn't seem to balance what it aims to help at...

Commit 1542199 from Dawid Weiss in branch 'dev/branches/branch_4x'
[ https://svn.apache.org/r1542199 ]

LUCENE-5283: allow top-level testcase filters.

Commit 1542201 from Dawid Weiss in branch 'dev/branches/lucene_solr_4_6'
[ https://svn.apache.org/r1542201 ]

LUCENE-5283: allow top-level testcase filters.

As far as I'm concerned, just aborting early would be enough. The "inconvenience"
of typing 'cd solr' is trivial, it was just a surprise that the behavior changed and
I worried that testing was broken.

There's no purpose served by making things complex just so I can "do what I've
always done". And I'm not about to insist that other people do work just so I
can be lazy.

So do what you think best. I always prefer ease of maintenance over just
having to adjust to a minor change in my behavior.

This isn't about you, Erick. If you've hit this, others will too. Let's leave it as is for now and see if there are any problems with the current patch.

Fixed on branch_4x, branch_36 and trunk.

Someone has changed something in the build system so that it no longer works (false alarms about no tests in a submodule).

All of this code is too fragile I think and should be removed. The value added is minimal and the headaches with maintenance are huge. The problem is that any subant or ant call which doesn't pass the required properties to detect the "top level" build will break it. I don't think there's a hook in Ant to allow detection of the top-level build file.

Please don't remove this! This catches me all the time.

Hang on, when did it break? I swear I've seen it working recently...

It doesn't. Or at least not for Solr. Try to filter for one class but run from the top level and it'll complain about zero tests.

I'll look into it again though. I'll see what's possible.

This feature caused a build "failure" for me when dataimporthandler-extras had all its tests skipped or ignored, when run as part of ant test-contrib from solr/:

-init-totals:

-test:

    [mkdir] Created dir: /var/lib/jenkins/jobs/Lucene-Solr-tests-5.2-Java8/workspace/solr/build/contrib/solr-dataimporthandler-extras/test
    [mkdir] Created dir: /var/lib/jenkins/jobs/Lucene-Solr-tests-5.2-Java8/workspace/solr/build/contrib/solr-dataimporthandler-extras/test/temp
    [mkdir] Created dir: /var/lib/jenkins/jobs/Lucene-Solr-tests-5.2-Java8/workspace/.caches/test-stats/solr-dataimporthandler-extras
   [junit4] <JUnit4> says hi! Master seed: 2085E7C0234F42B1
   [junit4] Executing 2 suites with 2 JVMs.
   [junit4] 
   [junit4] Started J0 PID(7435@goose).
   [junit4] Started J1 PID(7498@goose).
   [junit4] Suite: org.apache.solr.handler.dataimport.TestMailEntityProcessor
   [junit4] Completed [1/2] on J1 in 0.47s, 6 tests, 6 skipped
   [junit4] 
   [junit4] Suite: org.apache.solr.handler.dataimport.TestTikaEntityProcessor
   [junit4] Completed [2/2] on J0 in 2.02s, 9 tests, 9 skipped
   [junit4] 
   [junit4] JVM J0:     0.84 ..     3.80 =     2.96s
   [junit4] JVM J1:     1.08 ..     2.82 =     1.74s
   [junit4] Execution time total: 3.87 sec.
   [junit4] Tests summary: 2 suites, 15 tests, 15 ignored
     [echo] 5 slowest tests:
[junit4:tophints]   2.02s | org.apache.solr.handler.dataimport.TestTikaEntityProcessor
[junit4:tophints]   0.47s | org.apache.solr.handler.dataimport.TestMailEntityProcessor

-check-totals:

BUILD FAILED
/var/lib/jenkins/jobs/Lucene-Solr-tests-5.2-Java8/workspace/solr/build.xml:249: The following error occurred while executing this line:
/var/lib/jenkins/jobs/Lucene-Solr-tests-5.2-Java8/workspace/solr/common-build.xml:454: The following error occurred while executing this line:
/var/lib/jenkins/jobs/Lucene-Solr-tests-5.2-Java8/workspace/solr/common-build.xml:511: The following error occurred while executing this line:
/var/lib/jenkins/jobs/Lucene-Solr-tests-5.2-Java8/workspace/lucene/common-build.xml:1501: Not even a single test was executed (a typo in the filter pattern maybe?).
Total time: 5 minutes 12 seconds

Looks like -Dtests.ifNoTests=ignore didn't make it into the final implementation. I thought using -Dtests.totals.toplevel=false might work, but still fails for me when I purposely test a non-existent testcase.

Is there some way to not fail when no tests run in a module?

Hi Steve. This must be a regression again. The problem here is that it's very difficult to tell whether a module build is stand-alone or has been invoked by (some) parent ant process. In effect it's very difficult to tell whether there really was no tests or maybe it's just that tests have been filtered out for this particular module.

I'm tired of this because it's such a pain to maintain and at the same time there are no good solutions – either somebody will be complaining that all tests can be filtered out and not cause a failure or experience what you have experienced.

I'll see if there's a way to add this "tests.ifNoTests" property, but I don't see much sense to it – either the check for the total number of tests works or it doesn't. Why skip something that doesn't work anyway.

This seems to work for me as a temporary workaround.

cd solr
ant test-contrib -Dtests.class="Foo" -Dtests.ifNoTests=ignore

Feel free to commit this in if you need it, Steve.

Thanks Dawid! Saves me from having to hack in something like this myself.

I'll commit - this shouldn't affect anybody who doesn't use it.

Commit 1682550 from Steve Rowe in branch 'dev/trunk'
[ https://svn.apache.org/r1682550 ]

LUCENE-5283: Introduce new build property tests.ifNoTests - when its value is 'ignore', 'ant test' will succeed even when zero tests were executed.

Commit 1682552 from Steve Rowe in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1682552 ]

LUCENE-5283: Introduce new build property tests.ifNoTests - when its value is 'ignore', 'ant test' will succeed even when zero tests were executed. (merged trunk r1682550)

Commit 1682554 from Steve Rowe in branch 'dev/branches/lucene_solr_5_2'
[ https://svn.apache.org/r1682554 ]

LUCENE-5283: Introduce new build property tests.ifNoTests - when its value is 'ignore', 'ant test' will succeed even when zero tests were executed. (merged trunk r1682550)