[HBASE-15651] Add report-flakies.py to use jenkins api to get failing tests

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Hadoop Flags: Reviewed

Release Note

To find recent set of flakies, run the script added by this patch. Run it to get usage information passing -h:

{code}
$ ./dev-support/report-flakies.py -h
{code}

If you get the below:

{code}
$ python ./dev-support/report-flakies.py
Traceback (most recent call last):
  File "./dev-support/report-flakies.py", line 25, in <module>
    import requests
ImportError: No module named requests
{code}

... install the requests module:

{code}
$ sudo pip install requests
{code}

• Assignee: Apekshit Sharma
• Reporter: Apekshit Sharma
• Created: 14/Apr/16 05:53
• Updated: 23/Jun/22 19:02
• Resolved: 16/May/16 23:19

Description

So i have written this simple script (attached) which looks at history of the post commit build and outputs a list of flaky tests with some numbers.

Next steps:
1. Setup a jenkins job (say find-flaky-tests) to run this script daily. We can either directly pull these results into other jobs using curl on this job's artifacts, or commit the list of flaky test to repo (idk if it's possible to commit something from jenkins job).
We'll collect results from both post-commit job (to add new flakies) and flaky-tests job (to delete tests which are no more flaky).

2. Change pre-commit and post-commit jobs to ignore these tests using --exclude maven flag. Someone familiar with yetus might be able to do it easily.

3. Setup a new job (say flaky-tests) to run only these flaky tests.

Issue Links

• is related to: HBASE-15012 Flakies: Umbrella issue to hang current set of flakies from