Code coverage reports

• Key: LUCENE-721
• Type: New Feature
• Status: Resolved
• Resolution: Fixed
• Priority: Minor
• Assignee: Grant Ingersoll
• Reporter: Michael Busch
• Component/s: Other

Description

Hi all,

We should be able to measure the code coverage of our unit testcases. I believe it would be very helpful for the committers, if they could verify before committing a patch if it does not reduce the coverage.

Furthermore people could take a look in the code coverage reports to figure out where work needs to be done, i. e. where additional testcases are neccessary. It would be nice if we could add a page to the Lucene website showing the report, generated by the nightly build. Maybe you could add that to your preview page (LUCENE-707), Grant?

I attach a patch here that uses the tool EMMA to generate the code coverage reports. EMMA is a very nice open-source tool released under the CPL (same license as junit). The patch adds three targets to common-build.xml:

• emma-check: verifys if both emma.jar and emma_ant.jar are in the ant classpath
• emma-instrument: instruments the compiled code
• generate-emma-report: generates an html code coverage report

The following steps are neccessary in order to generate a code coverage report:

• add emma.jar and emma_ant.jar to your ant classpath (download emma from http://emma.sourceforge.net/)
• execute ant target 'emma-instrument' (depends on compile-test, so it will compile all core and test classes)
• execute ant target 'test' to run the unit tests
• execute ant target 'generate-emma-report'

To view the emma report open build/test/emma/index.html

Comments

The patch described above.

[Michael Busch]

I attach a code coverage report of the current HEAD (revision: 477535), so people can check out how a report looks like.

The current code coverage looks not bad: 77% of the code lines are being tested by our unit tests. However, the coverage is not perfect: e. g. the PorterStemmer or IndexWriter.addIndexes(IndexReader[]) are not tested at all.

[Otis Gospodnetic]

Nice. I think we can't include EMMA jars int he repository, though, so you'll want to add the ability to download the Jar on the fly, just like Grant did it for the benchmark stuff.

[Michael Busch]

Good idea, I could do that. But currently we don't fetch the JUnit jars either, just give a hint where to download them in case they cannot be found in the ant classpath. I did the same with the EMMA jars. Should we do it for both (JUnit and EMMA), just for EMMA or leave it like it is for now?

[Otis Gospodnetic]

Sounds like my previous comment was off - if we don't distribute EMMA jars and just require the person to have them, what you have will do. How about the nightly build, if we include this on the site? I'm not familiar with it enough to comment. Yonik and Doug will know if/how things can be set up there with EMMA jars, should we include this on the site.

[Grant Ingersoll]

Care to update this for Clover, Michael? Then I can hook it into the site and the nightly builds.

Thanks,
Grant

[Michael Busch]

Sure Grant, I will update the patch soon to use clover instead of EMMA. Great that you will add it to the website!

Thanks,
Michael

[Michael Busch]

Here it is, Grant. This new patch uses Clover to generate code coverage reports. Simply add clover.jar to the ant classpath, do a "clean" and run the target "test". During compiling Clover will automatically instrument all classes under src/java.

After the junit tests are done, execute the new target "generate-clover-reports". Clover will generate the html report in build/test/clover/reports.

Note: If clover.jar can not be found in the ant classpath then "test" will simply print an info message, saying that code coverage reports are disabled, and exeute the junit tests like it did before this patch.

[Grant Ingersoll]

I'm not familiar w/ clover, but here's what I did, when trying out your patch (which works):

1. applied the patch
2. Put clover.jar and clover.license (retrieved from the committers private repository into my ANT lib)
3. Ran "ant clover" – this passed and said clover is enabled
4. Ran "ant test"
5. ant generate-clover-reports
6. View the reports! They look good and I think will prove to be useful, especially for people who are looking for how they can help make Lucene better.

Is there anyway to turn off clover other than removing the jar? It seems like it takes quite a bit longer to run the tests and I was thinking I may just want to enable it when running nightly, but that most people wouldn't want it on by default even though they may have clover in there ANT lib.
I wanted to do something like adding a property named run.clover (off by default) that could then be overridden by -Drun.clover=true on the command line, but I admit I don't see in ANT how to AND together the clover.present attribute with this new property. The CONDITION and AVAILABLE tasks in ANT don't seem to allow for ANDing two existing properties, or perhaps I am missing something. I admit, though, my ANT knowledge isn't what it used to be since I moved to Maven a long time ago, so if someone can chime in here it would be appreciated.

Otherwise, the patch looks good and I will commit it once we figure out whether being able to turn it off is important to people and how to do it.

[Grant Ingersoll]

OK, I committed this, with a minor change so that it is off by default (thanks Erik, for the ANT info).

To run, you need three things:
1. clover.jar in ANT lib
2. clover.license in ANT lib
3. On the command line or in the your ANT properties, if you want it on all the time, you must specify -Drun.clover=true

Thanks, Micheal.

I will incorporate this into the nightly build per http://issues.apache.org/jira/browse/LUCENE-708

[Grant Ingersoll]

Committed the change. Linked to the reports from the Resources -> Developers page on the documentation. Current report was generated on 12/16/06. Once Lucene 708 is resolved, these reports will be updated nightly.

Thanks Michael!