Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Cannot Reproduce
-
None
-
None
-
None
-
None
Description
on a locally built 1.6.0-SNAPSHOT based on the SHA 417902e218c566333b6ea5ac492186ae305e5e16 (Dec 31st) alongside an apache hadoop 1.2.1 install.
I had a single node deployment on my laptop, which had probably been interrupted by my laptop sleep during some ingest.
When i restarted the tserver, I got a "FATAL: Garbage collection may be interfering with lock keep-alive. Halting." log message after some recovery attempts and the tserver shut down.
I restarted and had similar results twice. vines and I tried to attach a debugger to the tserver on startup the next time, and recovery completed without any failures! I have restarted the tserver since then with no issues.
The only notable thing special about my workspace (since it affects the startup paths) is that i'm using some custom Authorizer and Authenticator plugins in Accumulo.
Log excerpts:
============== FIRST TIME ============== 2014-01-09 12:42:58,241 [server.Accumulo] INFO : tserver starting ...snip... 2014-01-09 12:44:47,820 [tserver.Tablet] INFO : Starting Write-Ahead Log recovery for d<;7006d 2014-01-09 12:44:47,821 [tserver.TabletServer] INFO : Looking for hdfs://localhost:9000/accumulo160b/recovery/2dceae9f-98c2-4f0c-9c66-27f4ab52113a/finished 2014-01-09 12:44:47,822 [tserver.TabletServer] INFO : Looking for hdfs://localhost:9000/accumulo160b/recovery/61b9709a-c945-41ae-946c-fc6425f4c96a/finished 2014-01-09 12:44:47,823 [log.SortedLogRecovery] INFO : Looking at mutations from hdfs://localhost:9000/accumulo160b/recovery/2dceae9f-98c2-4f0c-9c66-27f4ab52113a for d<;7006d 2014-01-09 12:44:47,900 [log.SortedLogRecovery] INFO : Looking at mutations from hdfs://localhost:9000/accumulo160b/recovery/61b9709a-c945-41ae-946c-fc6425f4c96a for d<;7006d 2014-01-09 12:44:47,990 [log.SortedLogRecovery] INFO : Scanning for mutations starting at sequence number 2 for tid 110 2014-01-09 12:45:32,510 [util.Halt] FATAL: Garbage collection may be interfering with lock keep-alive. Halting. ============== SECOND TIME ============== 2014-01-09 13:55:17,890 [server.Accumulo] INFO : tserver starting ...snip... 2014-01-09 13:55:33,687 [log.LogSorter] INFO : Copying hdfs://localhost:9000/accumulo160b/wal/localhost+9997/7e36fa46-95df-4b58-a1e4-6a29f43843a8 to hdfs://localhost:9000/accumulo160b/recovery/7e36fa46-95df-4b58-a1e4-6a29f43843a8 2014-01-09 13:55:33,688 [log.LogSorter] INFO : Copying hdfs://localhost:9000/accumulo160b/wal/localhost+9997/e3428689-9011-4908-8990-9bd079978983 to hdfs://localhost:9000/accumulo160b/recovery/e3428689-9011-4908-8990-9bd079978983 2014-01-09 13:56:31,628 [util.Halt] FATAL: Garbage collection may be interfering with lock keep-alive. Halting. ============== THIRD TIME ============== 2014-01-09 14:04:16,315 [server.Accumulo] INFO : tserver starting ...snip... 2014-01-09 14:04:32,078 [log.LogSorter] INFO : Copying hdfs://localhost:9000/accumulo160b/wal/localhost+9997/e3428689-9011-4908-8990-9bd079978983 to hdfs://localhost:9000/accumulo160b/recovery/e3428689-9011-4908-8990-9bd079978983 2014-01-09 14:04:32,079 [log.LogSorter] INFO : Copying hdfs://localhost:9000/accumulo160b/wal/localhost+9997/a7467054-ec80-4734-974d-7c55c201b332 to hdfs://localhost:9000/accumulo160b/recovery/a7467054-ec80-4734-974d-7c55c201b332 2014-01-09 14:04:32,370 [util.NativeCodeLoader] WARN : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2014-01-09 14:04:32,811 [log.LogSorter] INFO : Finished log sort a7467054-ec80-4734-974d-7c55c201b332 11543822 bytes 1 parts in 732ms 2014-01-09 14:04:32,817 [log.LogSorter] INFO : Copying hdfs://localhost:9000/accumulo160b/wal/localhost+9997/7e36fa46-95df-4b58-a1e4-6a29f43843a8 to hdfs://localhost:9000/accumulo160b/recovery/7e36fa46-95df-4b58-a1e4-6a29f43843a8 2014-01-09 14:05:27,482 [util.Halt] FATAL: Garbage collection may be interfering with lock keep-alive. Halting.