[BEAM-7714] Allow retries of PostCommit test suites per Python version

• Type: Sub-task
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: testing
• Assignee: Valentyn Tymofieiev
• Reporter: Valentyn Tymofieiev
• Created: 09/Jul/19 22:10
• Updated: 16/May/20 13:30
• Resolved: 19/Jul/19 21:43

Description

Currently Python PostCommit test executes 4 tests running the set of tests under Python 2.7, 3.5-3.7. When test execution fails due to a flake, contributors have to rerun the whole suite. Having a possibility to re-run test suite only for a particular version of Python would make it easier to receive a green run.

Some considerations:

• increasing number of Jenkins job will increase the number of slots required by postcommit, this will slow down the queue, unless we increase number of slots. We can investigate utilization of Jenkins workers to see if slot increase is advisable.

• we could introduce phrase-only suites "Run Python 3.7 PostCommits", that will be separate jenkins jobs (1 suite, 1 slot) in addition to current jobs. phrase-only suites will not be triggered on the PR but will be triggered manually when users want to re-run tests for particular version. It may cause confusion on a PR though, since PR author will have to explain to reviewers that Python 3 Postcommit suite failed, but only 3.6 portion failed, and I re-ran Py3.6 portion only in this separate jenkins Job and it passed, so PR is safe to merge.

Issue Links

• links to: GitHub Pull Request #9093
• links to: GitHub Pull Request #9143