[RATIS-1677] Do not auto format RaftStorage in RECOVER

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: server
• Assignee: Tsz-wo Sze
• Reporter: Tsz-wo Sze
• Created: 17/Aug/22 17:17
• Updated: 10/Jan/24 21:03
• Resolved: 26/Aug/22 01:52

Description

A server should never create a new directory unless RaftStorage.StartupOption.FORMAT is specified. Otherwise, it will cause data loss since the raft log committed in that server is (temporarily) lost. Consider the following

1. Server A and Server B were fast and had commit index 100 while server C had commit index 90.
2. The disk of Server A was bad and it was restarted with RECOVER. Since it cannot access its directory in the bad disk, it created a new directory in another disk. Then, its commit index was reset and became -1.
3. Server C started a leader election and Server A voted for it. The commit index became 90 – it lost the commits from index 91 to index 100.

Issue Links

• breaks: RATIS-1694 Fix the compatibility issue of RATIS-1677
• is duplicated by: RATIS-1668 When an existing dir fails, server should not initialize a new dir
• relates to: RATIS-1995 Prevent data loss when a storage is accidentally re-formatted
• relates to: HDDS-7103 Ratis log storage directories unchecked causing unhandled exception on datanode restart
• relates to: RATIS-1871 Auto format RaftStorage when there is only one directory configured.
• links to: GitHub Pull Request #718