[APLO-282] org.iq80.leveldb.DBException - possible LevelDB store corruption?

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6
• Environment: apollo-99-trunk-20121105.032511-125
• Assignee: Hiram R. Chirino
• Reporter: Lionel Cons
• Created: 20/Dec/12 06:55
• Updated: 04/Feb/13 14:03
• Resolved: 28/Jan/13 18:57

Description

One of our broker refuses to start and logs:

2012-12-19 17:15:27,828 | INFO | Starting store: leveldb store at /var/lib/apollo/data |
2012-12-19 17:15:28,001 | INFO | Opening the log file took: 62.94 ms |
2012-12-19 17:15:28,214 | WARN | DB operation failed. (entering recovery mode): org.iq80.leveldb.DBException: IO error: /var/lib/apollo/data/dirty.index/016282.sst: No such file or directory | 13bb3f2b7c0
[...]
2012-12-19 17:15:32,863 | INFO | broker startup is waiting on start virtual-host: apollo |

and it hangs there.

Here is the stack trace:

2012-12-19 17:15:28,217 | WARN | stackref=13bb3f2b7c0
org.iq80.leveldb.DBException: IO error: /var/lib/apollo/data/dirty.index/016282.sst: No such file or directory
at org.fusesource.leveldbjni.internal.JniDB.get(JniDB.java:78)
at org.apache.activemq.apollo.broker.store.leveldb.HelperTrait$RichDB.get(HelperTrait.scala:113)
at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.org$apache$activemq$apollo$broker$store$leveldb$LevelDBClient$$load_log_refs(LevelDBClient.scala:595)
at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$start$6.apply$mcV$sp(LevelDBClient.scala:356)
at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$start$6.apply(LevelDBClient.scala:332)
at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$start$6.apply(LevelDBClient.scala:332)
at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.retry(LevelDBClient.scala:714)
at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.start(LevelDBClient.scala:332)
at org.apache.activemq.apollo.broker.store.leveldb.LevelDBStore$$anonfun$_start$4.apply$mcV$sp(LevelDBStore.scala:114)
at org.fusesource.hawtdispatch.package$$anon$3.run(hawtdispatch.scala:357)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/lib/apollo/data/dirty.index/016282.sst: No such file or directory
at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
at org.fusesource.leveldbjni.internal.NativeDB.get(NativeDB.java:307)
at org.fusesource.leveldbjni.internal.NativeDB.get(NativeDB.java:300)
at org.fusesource.leveldbjni.internal.NativeDB.get(NativeDB.java:293)
at org.fusesource.leveldbjni.internal.JniDB.get(JniDB.java:73)
... 12 more