Per one of the TODOs in Journal.java, there is currently a lack of atomicity in the acceptRecovery() code path. In particular, we have the following actions executed non-atomically:
- Download a new edits_inprogress_N from some other node
- Persist the paxos recovery file to disk.
If the JN crashes between these two steps, then we may be left in the state whereby the edits_inprogress file has different data than the Paxos data left over on the disk from a previous recovery attempt. This causes the next prepareRecovery() to fail with an AssertionError.
I discovered this by randomly injecting a fault between the two steps, and then running the randomized fault test on a cluster. This resulted in some AssertionErrors in the test logs.