Accumulo
  1. Accumulo
  2. ACCUMULO-2505

IllegalTableTransitionException while attempting to drop a table

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 1.5.1
    • Fix Version/s: None
    • Component/s: master
    • Labels:
      None
    • Environment:

      ec2 single node instance

      Description

      Saw this while attempting a series of table deletes on a ec2 setup (i.e. minimal resources)

      java.lang.RuntimeException: org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
              at org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:150)
              at org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:232)
              at org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:215)
              at org.apache.accumulo.server.master.tableOps.TraceRepo.call(TraceRepo.java:65)
              at org.apache.accumulo.fate.Fate$TransactionRunner.run(Fate.java:64)
              at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:34)
              at java.lang.Thread.run(Thread.java:662)
      Caused by: org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException
              at org.apache.accumulo.server.master.state.tables.TableManager$1.mutate(TableManager.java:143)
              at org.apache.accumulo.fate.zookeeper.ZooReaderWriter.mutate(ZooReaderWriter.java:134)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.accumulo.server.zookeeper.ZooReaderWriter$1.invoke(ZooReaderWriter.java:68)
              at com.sun.proxy.$Proxy10.mutate(Unknown Source)
              at org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:118)
              ... 6 more
      

        Issue Links

          Activity

          John Vines created issue -
          Hide
          John Vines added a comment -
           2014-03-17 17:37:26,674 [thrift.ProcessFunction] ERROR: Internal error processing waitForTableOperation
          java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.10.1.85:9997
                  at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:174)
                  at org.apache.accumulo.server.master.tableOps.CleanUp.isReady(DeleteTable.java:97)
                  at org.apache.accumulo.server.master.tableOps.CleanUp.isReady(DeleteTable.java:55)
                  at org.apache.accumulo.server.master.tableOps.TraceRepo.isReady(TraceRepo.java:50)
                  at org.apache.accumulo.fate.Fate$TransactionRunner.run(Fate.java:61)
                  at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:34)
                  at java.lang.Thread.run(Thread.java:662)
          Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.10.1.85:9997
                  at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:712)
                  at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:613)
                  at org.apache.accumulo.core.client.impl.MetadataLocationObtainer.lookupTablets(MetadataLocationObtainer.java:181)
                  at org.apache.accumulo.core.client.impl.TabletLocatorImpl.processInvalidated(TabletLocatorImpl.java:595)
                  at org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:539)
                  at org.apache.accumulo.core.client.impl.TabletLocator$1._locateTablet(TabletLocator.java:116)
                  at org.apache.accumulo.core.client.impl.TabletLocatorImpl.locateTablet(TabletLocatorImpl.java:377)
                  at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:216)
                  at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:82)
                  at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:164)
                  ... 6 more
          Caused by: org.apache.thrift.TApplicationException: Internal error processing startMultiScan
                  at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
                  at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
                  at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startMultiScan(TabletClientService.java:294)
                  at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startMultiScan(TabletClientService.java:274)
                  at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:652)
                  ... 15 more
          

          That seems to be the initial error that caused this. I also noticed that the tserver was still reporting splits for this tablet after it was logged that the delete table opreration had started.

          Show
          John Vines added a comment - 2014-03-17 17:37:26,674 [thrift.ProcessFunction] ERROR: Internal error processing waitForTableOperation java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.10.1.85:9997 at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:174) at org.apache.accumulo.server.master.tableOps.CleanUp.isReady(DeleteTable.java:97) at org.apache.accumulo.server.master.tableOps.CleanUp.isReady(DeleteTable.java:55) at org.apache.accumulo.server.master.tableOps.TraceRepo.isReady(TraceRepo.java:50) at org.apache.accumulo.fate.Fate$TransactionRunner.run(Fate.java:61) at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:34) at java.lang. Thread .run( Thread .java:662) Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server 10.10.1.85:9997 at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:712) at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:613) at org.apache.accumulo.core.client.impl.MetadataLocationObtainer.lookupTablets(MetadataLocationObtainer.java:181) at org.apache.accumulo.core.client.impl.TabletLocatorImpl.processInvalidated(TabletLocatorImpl.java:595) at org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:539) at org.apache.accumulo.core.client.impl.TabletLocator$1._locateTablet(TabletLocator.java:116) at org.apache.accumulo.core.client.impl.TabletLocatorImpl.locateTablet(TabletLocatorImpl.java:377) at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:216) at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:82) at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:164) ... 6 more Caused by: org.apache.thrift.TApplicationException: Internal error processing startMultiScan at org.apache.thrift.TApplicationException.read(TApplicationException.java:108) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71) at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startMultiScan(TabletClientService.java:294) at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startMultiScan(TabletClientService.java:274) at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:652) ... 15 more That seems to be the initial error that caused this. I also noticed that the tserver was still reporting splits for this tablet after it was logged that the delete table opreration had started.
          Hide
          John Vines added a comment -

          Which traces to this-

          2014-03-17 17:37:26,536 [tabletserver.TabletServer] WARN : exception while doing multi-scan 
          java.lang.NullPointerException
                  at org.apache.accumulo.server.conf.TableConfiguration.get(TableConfiguration.java:122)
                  at org.apache.accumulo.server.conf.TableConfiguration.get(TableConfiguration.java:110)
                  at org.apache.accumulo.core.conf.AccumuloConfiguration.getBoolean(AccumuloConfiguration.java:129)
                  at org.apache.accumulo.server.tabletserver.FileManager$ScanFileManager.<init>(FileManager.java:449)
                  at org.apache.accumulo.server.tabletserver.FileManager.newScanFileManager(FileManager.java:547)
                  at org.apache.accumulo.server.tabletserver.TabletServerResourceManager$TabletResourceManager.newScanFileManager(TabletServerResourceManager.java:500)
                  at org.apache.accumulo.server.tabletserver.Tablet$ScanDataSource.createIterator(Tablet.java:2068)
                  at org.apache.accumulo.server.tabletserver.Tablet$ScanDataSource.iterator(Tablet.java:2047)
                  at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.seek(SourceSwitchingIterator.java:163)
                  at org.apache.accumulo.server.tabletserver.Tablet.lookup(Tablet.java:1623)
                  at org.apache.accumulo.server.tabletserver.Tablet.lookup(Tablet.java:1730)
                  at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$LookupTask.run(TabletServer.java:1038)
                  at org.apache.accumulo.server.tabletserver.TabletServerResourceManager.executeReadAhead(TabletServerResourceManager.java:723)
                  at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueMultiScan(TabletServer.java:1338)
                  at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startMultiScan(TabletServer.java:1307)
                  at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                  at java.lang.reflect.Method.invoke(Method.java:597)
                  at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:63)
                  at com.sun.proxy.$Proxy10.startMultiScan(Unknown Source)
                  at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startMultiScan.getResult(TabletClientService.java:1985)
                  at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startMultiScan.getResult(TabletClientService.java:1969)
                  at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
                  at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
          

          Which is another manifestation of accumulo-2489

          Show
          John Vines added a comment - Which traces to this- 2014-03-17 17:37:26,536 [tabletserver.TabletServer] WARN : exception while doing multi-scan java.lang.NullPointerException at org.apache.accumulo.server.conf.TableConfiguration.get(TableConfiguration.java:122) at org.apache.accumulo.server.conf.TableConfiguration.get(TableConfiguration.java:110) at org.apache.accumulo.core.conf.AccumuloConfiguration.getBoolean(AccumuloConfiguration.java:129) at org.apache.accumulo.server.tabletserver.FileManager$ScanFileManager.<init>(FileManager.java:449) at org.apache.accumulo.server.tabletserver.FileManager.newScanFileManager(FileManager.java:547) at org.apache.accumulo.server.tabletserver.TabletServerResourceManager$TabletResourceManager.newScanFileManager(TabletServerResourceManager.java:500) at org.apache.accumulo.server.tabletserver.Tablet$ScanDataSource.createIterator(Tablet.java:2068) at org.apache.accumulo.server.tabletserver.Tablet$ScanDataSource.iterator(Tablet.java:2047) at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.seek(SourceSwitchingIterator.java:163) at org.apache.accumulo.server.tabletserver.Tablet.lookup(Tablet.java:1623) at org.apache.accumulo.server.tabletserver.Tablet.lookup(Tablet.java:1730) at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler$LookupTask.run(TabletServer.java:1038) at org.apache.accumulo.server.tabletserver.TabletServerResourceManager.executeReadAhead(TabletServerResourceManager.java:723) at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.continueMultiScan(TabletServer.java:1338) at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.startMultiScan(TabletServer.java:1307) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:63) at com.sun.proxy.$Proxy10.startMultiScan(Unknown Source) at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startMultiScan.getResult(TabletClientService.java:1985) at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startMultiScan.getResult(TabletClientService.java:1969) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) Which is another manifestation of accumulo-2489
          John Vines made changes -
          Field Original Value New Value
          Link This issue duplicates ACCUMULO-2489 [ ACCUMULO-2489 ]
          John Vines made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              John Vines
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development