Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Cannot Reproduce
-
None
-
None
-
None
Description
While running the random walk test on a 10 node cluster, the shard test failed with the following errors.
07 23:43:38,483 [shard.CloneIndex] DEBUG: Cloned ST_index_26192_1328649015908_tmp from ST_index_26192_1328649015908 flush: 5825ms clone: 96ms
07 23:43:38,503 [impl.ThriftScanner] DEBUG: Failed to locate tablet for table : rr row :
07 23:49:52,795 [impl.ThriftScanner] DEBUG: Scan failed, not serving tablet (ln<<,xxx.xxx.xxx.10:9997)
07 23:51:11,539 [shard.VerifyIndex] DEBUG: Verified 65438656 index entries
07 23:51:12,052 [shard.Search] DEBUG: Looking up terms [67x, 1o3g, dl4, e4m] expect to find 2682000000000000
07 23:51:12,110 [impl.TabletServerBatchReaderIterator] DEBUG: Server : xxx.xxx.xxx.8:9997 msg : null
ThriftSecurityException(user:root, code:PERMISSION_DENIED)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$startMultiScan_result.read(TabletClientService.java:7586)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startMultiScan(TabletClientService.java:306)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startMultiScan(TabletClientService.java:274)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:83)
at $Proxy2.startMultiScan(Unknown Source)
at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:539)
at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:335)
at cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
07 23:51:12,113 [impl.TabletServerBatchReaderIterator] DEBUG: Error PERMISSION_DENIED - User does not have permission to perform this action
org.apache.accumulo.core.client.AccumuloSecurityException: Error PERMISSION_DENIED - User does not have permission to perform this action
at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:576)
at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:335)
at cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: ThriftSecurityException(user:root, code:PERMISSION_DENIED)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$startMultiScan_result.read(TabletClientService.java:7586)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startMultiScan(TabletClientService.java:306)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startMultiScan(TabletClientService.java:274)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:83)
at $Proxy2.startMultiScan(Unknown Source)
at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:539)
... 5 more
07 23:51:12,114 [randomwalk.Module] DEBUG: Properties for node: shard.Search
07 23:51:12,114 [randomwalk.Module] DEBUG: teardown:
07 23:51:12,114 [randomwalk.Module] DEBUG: maxSec:
07 23:51:12,114 [randomwalk.Module] DEBUG: maxHops:
07 23:51:12,114 [randomwalk.Module] DEBUG: Properties for node: Shard.xml
07 23:51:12,114 [randomwalk.Module] DEBUG: teardown:
07 23:51:12,114 [randomwalk.Module] DEBUG: maxSec:
07 23:51:12,114 [randomwalk.Module] DEBUG: maxHops:
07 23:51:12,114 [randomwalk.Framework] ERROR: Error during random walk
java.lang.Exception: Error running node Shard.xml
at org.apache.accumulo.server.test.randomwalk.Module.visit(Module.java:259)
at org.apache.accumulo.server.test.randomwalk.Framework.run(Framework.java:61)
at org.apache.accumulo.server.test.randomwalk.Framework.main(Framework.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.accumulo.start.Main$1.run(Main.java:89)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.Exception: Error running node shard.Search
at org.apache.accumulo.server.test.randomwalk.Module.visit(Module.java:259)
at org.apache.accumulo.server.test.randomwalk.Module.visit(Module.java:251)
... 8 more
Caused by: org.apache.accumulo.core.client.TableDeletedException
at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:357)
at cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
... 1 more
After seeing this looked and saw that the table existed. So for some reason a tablet server and client process did not see metadata related to the table. This happened after a delete and rename. One possibility is that the client is using the old table id from before the delete and rename. However, not sure how this could have happened. If the log messages printed the table id this would be easy to verify. Another possibility is that its some sort of zookeeper hiccup.