[BOOKKEEPER-237] Automatic recovery of under-replicated ledgers and its entries

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 4.0.0, 4.1.0
• Component/s: bookkeeper-auto-recovery
• Assignee: Rakesh Radhakrishnan
• Reporter: Rakesh Radhakrishnan
• Created: 04/May/12 12:37
• Updated: 14/Jan/13 14:23
• Resolved: 04/Jan/13 11:59

Description

As per the current design of BookKeeper, if one of the BookKeeper server dies, there is no automatic mechanism to identify and recover the under replicated ledgers and its corresponding entries. This would lead to losing the successfully written entries, which will be a critical problem in sensitive systems. This document is trying to describe few proposals to overcome these limitations.

Sub-Tasks

• Recording of underreplication of ledger entries: Closed
• Detection of under replication: Closed
• Rereplicating of under replicated data: Closed
• Provide automatic mechanism to know bookie failures: Closed
• Ability to disable auto recovery temporarily: Closed
• bookkeeper does not put enough meta-data in to do recovery properly: Closed
• Periodic checking of ledger replication status: Closed
• Provide LedgerFragmentReplicator which should replicate the fragments found from LedgerChecker: Closed
• Prepare bookie vs ledgers cache and will be used by the Auditor: Closed
• Provide distributed lock implementation which will be used by Replication worker while replicating fragments.: Resolved
• Exceptions for replication: Closed
• Manage auditing and replication processes: Closed
• Delay the replication of a ledger if RW found that its last fragment is in underReplication.: Closed
• Document about Auto replication service in BK: Closed
• LedgerManagers should consider 'underreplication' node as a special Znode: Closed
• ReplicationWorker may not get ZK watcher notification on UnderReplication ledger lock deletion.: Closed
• ZkLedgerUnderreplicationManager.markLedgerUnderreplicated() is adding duplicate missingReplicas while multiple bk failed for the same ledger: Closed
• Clean up LedgerManagerFactory and LedgerManager usage in tests: Closed
• replicateLedgerFragment should throw Exceptions in error conditions: Closed
• It should not be possible to replicate a ledger fragment which is at the end of an open ledger: Closed
• Ledger entries should be replicated sequentially instead of parallel.: Closed
• Let's add Thread name for ReplicationWorker thread.: Closed
• Integration Test - Perform bookie rereplication cycle by Auditor-RW processes: Closed
• LedgerChecker returns underreplicated fragments for an closed ledger with no entries: Closed
• Hierarchical zk underreplication manager should clean up its hierarchy when done to allow for fast acquisition of underreplicated entries: Closed
• Store hostname of locker in replication lock: Closed
• SingleFragmentCallback should be created with the fragment first entry id, not the first stored id: Resolved
• Lock does not guarantee any access order and not giving chance to longest-waiting RW: Resolved
• Make auditor Vote znode store a protobuf containing the host that voted: Closed
• Expose command options in bookie scripts to disable/enable auto recovery temporarily: Closed
• Provide an option to start Autorecovery along with Bookie Servers: Closed
• Ensure that the auditor and replication worker will shutdown if they lose their ZK session: Closed