[HDDS-4639] Safemode wait may end without checking

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Ozone CLI
• Labels: pull-request-available
• Target Version/s: 1.1.0
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 04/Jan/21 10:02
• Updated: 07/Jan/21 14:17
• Resolved: 07/Jan/21 10:57

Description

Intermittent failure in secure acceptance test:

output

Wait for safemode exit                                                | FAIL |
255 != 0

log

Running command 'ozone admin safemode wait -t 2 2>&1'.
${rc} = 255
${output} = WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.1.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Safe mode is not ended within the timeout period.

This is executed with safemode already OFF. Also, notice that neither SCM is out of safe mode, nor SCM is in safe mode is shown.

It indicates that ozone admin safemode wait may conclude without actually checking status, if it takes more time to create SCM client than the given timeout:

https://github.com/apache/ozone/blob/aacba62a1a1bdee45dad15e82ef64019ff67b767/hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/SafeModeWaitSubcommand.java#L60-L62

    while (getRemainingTimeInSec() > 0) {
      try (ScmClient scmClient = scmOption.createScmClient()) {
        while (getRemainingTimeInSec() > 0) {

It is reproducible by adding sleep in createScmClient().

Issue Links

• links to: GitHub Pull Request #1751