[FLINK-29198] RetryExtension doesn't make the test fail if the retries are exhausted

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0, 1.16.0, 1.17.0
• Fix Version/s: 1.17.0, 1.15.3, 1.16.1
• Component/s: Tests
• Labels: pull-request-available starter
• Assignee: Ryan Skraba
• Reporter: Matthias Pohl
• Created: 05/Sep/22 13:30
• Updated: 04/Nov/22 15:40
• Resolved: 04/Nov/22 13:24

Description

FLINK-24627 introduced retry functionality for JUnit5-based tests. It appears that the retry mechanism doesn't have the desired behavior: If the retries are exhausted without the test ever succeeding will result in the test being ignored. I would expect the test to fail in that case. Otherwise, a CI run would succeed without anyone noticing the malfunctioning of the test.

Issue Links

• causes: FLINK-29891 The RetryRule tests print some excessive stacktraces that might confuse people investigating actual issues
• is caused by: FLINK-24627 Port JUnit 4 rules to JUnit5 extensions
• Testing discovered: FLINK-29245 [JUnit 5 Migration] Remove RetryRule
• links to: GitHub Pull Request #20805