[HBASE-27425] Run flaky test job more often

• Type: Task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.6.0, 3.0.0-alpha-4, 2.5.2, 2.4.16
• Component/s: test
• Hadoop Flags: Reviewed
• Assignee: Sean Busbey
• Reporter: Sean Busbey
• Created: 12/Oct/22 18:26
• Updated: 24/Oct/22 20:53
• Resolved: 21/Oct/22 06:35

Description

We currently run the flaky tests once per 4 hours and keep a backlog of 50 runs. That means it takes a perfect test a bit over 8 days to get back out of being penalized.

we regularly are leaving a bunch of executors idle (see attached PNG) and I think we should run the tests more often.

we use an executor per branch for a flaky test run. the duration depends on the specific tests that are flagged, but it has a timeout of 2 hours.

as an example the master branch tests currently take < 30 minutes (see other attached PNG).

Issue Links

• links to: GitHub Pull Request #4828