[APLO-293] Apollo should try to recover messages from a corrupted store

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Do
• Fix Version/s: 1.8
• Environment: apollo-99-trunk-20130202.135855-180
• Assignee: Hiram R. Chirino
• Reporter: Lionel Cons
• Created: 04/Feb/13 14:01
• Updated: 06/Apr/22 19:16
• Resolved: 06/Apr/22 19:16

Description

Due to other bugs (mainly APLO-257 but not only), we sometimes have to kill Apollo the hard way as it would not stop gracefully.

This almost always leaves the LevelDB store corrupted. When (re)starting, we see messages like:

2013-02-03 12:47:13,099 | WARN | DB operation failed. (entering recovery mode): org.iq80.leveldb.DBException: IO error: /var/lib/apollo/data/dirty.index/001869.sst: No such file or directory | 13c9fe18242

(see also APLO-282)

At this point Apollo hangs. The only solution is to kill it once more and completely destroy the message store, loosing all messages

Could Apollo try to recover at least some messages in these situations instead of hanging during startup?