[NUTCH-2771] Tests in nightly builds: speed up long runners

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.16
• Fix Version/s: 1.21
• Component/s: build, test
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 27/Feb/20 09:11
• Updated: 30/Mar/24 17:19

Description

The Nutch tests run by "ant test" or "ant nightly") take rather long to run. Although all tests are implemented as JUnit tests, some tests are more integration tests, eg. launching a Jetty web server and fetching documents from it. It's nice to have also higher level tests, and they are expected to long runner than a simple unit test. However, some of the test classes take really long to run (times taken from https://builds.apache.org/job/Nutch-trunk/3663/consoleText):

    [junit] Running org.apache.nutch.segment.TestSegmentMergerCrawlDatums
    [junit] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 133.898 sec

    [junit] Running org.apache.nutch.segment.TestSegmentMerger
    [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.026 sec

    [junit] Running org.apache.nutch.crawl.TestGenerator
    [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.03 sec

    [junit] Running org.apache.nutch.fetcher.TestFetcher
    [junit] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.805 sec

    [junit] Running org.apache.nutch.urlfilter.fast.TestFastURLFilter
    [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.36 sec

    [junit] Running org.apache.nutch.parse.tika.TestPdfParser
    [junit] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.974 sec

    [junit] Running org.apache.nutch.parse.tika.TestImageMetadata
    [junit] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.113 sec

    [junit] Running org.apache.nutch.parse.feed.TestFeedParser
    [junit] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.369 sec

    [junit] Running org.apache.nutch.crawl.TestInjector
    [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.15 sec

We could try to speed up at least some of these long-running tests.