Details
Major Description
The HTTP protocol plugin unit tests may fail at random. The reasons are:
- the unit tests run the test web server delivering test pages using the same port (inherited from AbstractHttpProtocolPluginTest)
- the plugin unit tests are executed in parallel (two concurrent threads)
From time to time two tests try to launch a web server listening on the same port. This consequently causes a failure, e.g. in a GitHub workflow:
[junit] Tests run: 14, Failures: 1, Errors: 0, Skipped: 4, Time elapsed: 4.735 sec [junit] Test org.apache.nutch.protocol.okhttp.TestBadServerResponses FAILED
The error message in (TEST-org.apache.nutch.protocol.http.TestBadServerResponses.txt, from a local test run):
2024-09-06 08:36:32,549 INFO o.a.n.p.AbstractHttpProtocolPluginTest [Thread-3] Socket on port 47505 closed: Address already in use (Bind failed) 2024-09-06 08:36:32,550 INFO o.a.n.p.AbstractHttpProtocolPluginTest [Thread-2] Socket on port 47505 closed: Socket closed 2024-09-06 08:36:32,599 INFO o.a.n.p.AbstractHttpProtocolPluginTest [main] Fetching http://127.0.0.1:47505/ 2024-09-06 08:36:32,600 ERROR o.a.n.p.h.Http [main] Failed to get protocol output java.net.ConnectException: Connection refused (Connection refused)
Possible solutions:
1. do not run plugin unit tests in parallel.
– Note: the parallelism does not save a lot of time. A test run on my laptop: 6'19'' (2 threads, failed) vs. 6'55'' (1 thread, success)
2. override the port in each unit test and
– ensure that a unique port number is used. Note: manually assigning unique numbers is difficult to maintain when new tests are added.
– or choose a random port number (making collisions unlikely)
3. try another port if one is already in use
Attachments
Issue Links
- links to