Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
The Ptest framework has a lot of usability issues some of which are listed below:
1. Ptest can run only one job at a time.
2. The pending queue resides on the pre-commit jenkins server. If the jenkins server is restarted we lose the queue and the devs don't understand why their patches were not run.
3. Average Ptest run takes about ~100 min which not bad considering it is running > 10k tests; many of which are very complex queries. We should look to see if we can further reduce the turn-around time.
Some of the ideas to improve the current state of Ptest could be:
1. Move the queue to the Ptest server so that it is not lost due to jenkins restarts.
2. The jenkins server could do some useful work instead of just waiting for Ptest server to return. I propose it should run some pre-checkin (for lack of a better word) tests which are reliable and which run fast. The advantage of having pre-checkin tests if there are issues with the patch, the pre-commit fails fast without the long turn-around time and devs will get a quick feedback on issues which need to be fixed in the patch before the full suite of tests need to be run. The second advantage is Ptest server will be running a fairly tested patches so hopefully will be doing useful work rather. This would also reduce load on the Ptest.
3. If Ptest server has a database we can do interesting analysis on the test results like identifying flaky tests automatically, generating weekly reports about the test status.
4. Have a web-interface of the Ptest server so that devs can check the status of the queue and which patch is being run currently.