[OAK-7866] Add tooling for recovering from an outdated journal

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.10.0, 1.9.13
• Component/s: segment-tar
• Assignee: Francesco Mari
• Reporter: Francesco Mari
• Created: 29/Oct/18 14:02
• Updated: 14/Dec/18 11:30
• Resolved: 04/Dec/18 13:39

Description

Some error conditions in the Segment Store might prevent the journal from being updated. After a restart the system will be rolled back to the latest version persisted in the journal. Every change performed after the system stopped updating the journal appears to be lost.

Unless some deep inconsistencies are found in the segments, it might be possible to recover from an outdated journal by performing the following steps:

• Backup the previous journal.
• Scan the segments for node records representing super-roots.
• Order the found records by segment creation date (ascending) and record number (ascending).
• Create a new journal with the ordered entries.
• Perform a sanity check of the journal (as the "check" tool does).

We should have a tool executing these steps. Being a "porcelain" tool, it should have a small and stable command-line interface, be safe in the face of unexpected error conditions, and be friendly when communicating to the user both in case of success and in case of errors.

Issue Links

• Blocked: OAK-7234 Check for outdated journal at startup
• is blocked by: OAK-7918 Extract checking logic from ConsistencyChecker
• is duplicated by: OAK-5885 segment-tar should have a tarmkrecovery command