Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.5.1
    • Fix Version/s: 1.5.2, 1.6.0
    • Component/s: master
    • Labels:
    • Environment:

      nightly single-node test

      Description

      testTrace in ConditionalWriterIT failed:

      java.lang.Exception: test timed out after 60000 milliseconds
      	at java.lang.Thread.sleep(Native Method)
      	at org.apache.accumulo.core.util.UtilWaitThread.sleep(UtilWaitThread.java:26)
      	at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:240)
      	at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:84)
      	at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:177)
      	at org.apache.accumulo.core.trace.TraceDump.printTrace(TraceDump.java:133)
      	at org.apache.accumulo.test.ConditionalWriterIT.testTrace(ConditionalWriterIT.java:1231)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:622)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
      

      Examining the logs, it seems like the client was having a hard time locating the trace table (table id 9):

      2014-03-07 04:33:40,445 [impl.ThriftScanner] TRACE: Failed to locate tablet for table : 9 row : 9dd07d1bd8b9bc0d
      ...
      2014-03-07 04:33:40,551 [impl.ThriftScanner] TRACE: Failed to locate tablet for table : 9 row : 9dd07d1bd8b9bc0d
      

      Examining the master logs, I see the trace table being created, assigned, and hosted.

      This test deletes everything in the trace table before tracing an operation:

          conn.tableOperations().deleteRows("trace", null, null);
      

      The master takes the table offline and re-writes the metadata table information so that it no longer has any data.

      The master then attempts to put the tablet online, but the operation fails:

      2014-03-07 04:33:40,079 [master.Master] ERROR: host1:47630 reports assignment failed for tablet 9<<
      

      The tablet server gives the real problem:

      2014-03-07 04:33:40,074 [tserver.TabletServer] WARN : exception trying to assign tablet 9<< file:////local/disk1/jenkins/workspace/accumulo16/test/target/mini-tests/org.apache.accumulo.test.functional.SimpleMacIT/1394184783304_4048/accumulo/tables/9/default_tablet
      java.lang.IllegalArgumentException: Time type unknown : ^@0
              at org.apache.accumulo.server.tablets.TabletTime.getInstance(TabletTime.java:67)
              at org.apache.accumulo.tserver.Tablet.<init>(Tablet.java:1296)
              at org.apache.accumulo.tserver.Tablet.<init>(Tablet.java:1211)
              at org.apache.accumulo.tserver.Tablet.<init>(Tablet.java:1067)
              at org.apache.accumulo.tserver.Tablet.<init>(Tablet.java:1056)
              at org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.java:2911)
              at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
              at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
              at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
              at java.lang.Thread.run(Thread.java:701)
      

      Tablet time should always be "L" or "M".

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ecn Eric Newton
                Reporter:
                ecn Eric Newton
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: