[CASSANDRA-16655] cassandra-cqlsh-tests.sh should return different return codes on circle vs jenkins

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 4.0-rc2, 4.0, 4.1-alpha1, 4.1
• Component/s: CI
• Bug Category: Correctness - Test Failure
• Severity: Critical
• Complexity: Low Hanging Fruit
• Discovered By: Unit Test
• Platform: All
• Since Version: 4.0-beta4

Source Control Link

https://github.com/apache/cassandra/commit/b67d04b6fe311d8c923d125c722e7a094a921f0c

• Assignee: Unassigned
• Reporter: Berenguer Blasi

Authors

Berenguer Blasi, Michael Semb Wever

Reviewers

Michael Semb Wever

• Created: 05/May/21 09:51
• Updated: 27/May/22 19:25
• Resolved: 05/May/21 16:11

Description

Recent jenkins runs appear broken (red) bc of a cqlsh test failure. That reports the whole run as broken instead of that specific step with failures and an unstable (yellow) run.

It appears the reason may be this change. At that time that was necessary to make circle fail that step. Otherwise cqlsh failures would be ignored and that step rendered green regarless making it useless. But it has been found now this makes jenkins report the whole run red upon a cqlsh test failure.

The solution seems to be to return different codes depending on whether we're on jenkins or circle. mck has suggested sthg along these lines:

# circleci wants non-zero exit to report failures, jenkins wants zero exit for a unstable status
[[ command -v circleci >/dev/null 2>&1 ]] && exit ${RETURN}
exit 0

Procedure:

• Fix the sh script
• Break a cqlsh test
• Test the test gets reported as a failure in circle
• Test the test gets reported as a failure in jenkins and that the run and step are yellow (not red)
• Fix the test
• Both circle and jenkins report green for that step

Issue Links

• is caused by: CASSANDRA-16121 Circleci should run cqlshlib tests as well
• links to: CASSANDRA-16655 PR trunk