[RATIS-1332] Ratis server couln't be recovered from failed initialization state

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: server
• Assignee: Janus Chow
• Reporter: Marton Elek
• Created: 09/Mar/21 11:37
• Updated: 15/May/23 16:30
• Resolved: 15/Mar/21 01:50

Description

I found this problem during the test of ratis 2.0.0-rc3 and earlier.

I noticed that in some cases the Ozone Manager (with ratis enabled true) couldn't be started any more (see HDDS-4703 for details).

After some investigation I found the following problem:

1. Ratis server initialized BEFORE om RPC (OzoneManager.startRpcServer)
2. If the RPC server is failed (due to missing DNS for example) the Ratis server is stopped during the initialization
3. AtomicOutputStream can leave some tmp files behind (like raft-meta.tmp, if it's not yet renamed)
4. After DNS problem is fixed the OM couldn't be started anymore as RaftStorageImpl.analyzeAndRecoverStorage requires FORMATTED or empty (!!!) directory. Directory with leftover tmp file is not empty.

  private StorageState analyzeAndRecoverStorage(boolean toLock) throws IOException {
    StorageState storageState = storageDir.analyzeStorage(toLock);
    if (storageState == StorageState.NORMAL) {
        // ...
    } else if (storageState == StorageState.NOT_FORMATTED &&
        storageDir.isCurrentEmpty()) {
     //never called this if one .tmp file exists from the previous attempts
      format();
      return StorageState.NORMAL;
    } else {
      return storageState;
    }
  }

The problem is that `cleanMetaTmpFile();` is called only in the first branch, but before checking if the directory is empty or not...

Issue Links

• fixes: HDDS-4703 New OM couldn't be started due to NOT_FORMATTED Ratis dir
• links to: GitHub Pull Request #442