[ACCUMULO-271] WholeRowIterator may call hasTop on unseeked source

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Labels: 14_qa_bug
• Assignee: Keith Turner
• Reporter: Keith Turner
• Created: 05/Jan/12 22:38
• Updated: 09/Feb/12 22:26
• Resolved: 10/Jan/12 17:10

Description

A random walker failed while running the security test against 1.4. Saw the following :

05 21:11:02,632 [randomwalk.Framework] ERROR: Error during random walk
java.lang.Exception: Error running node Security.xml
  .
  . 
  .
Caused by: org.apache.thrift.TApplicationException: Internal error processing waitForTableOperation
        at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
        at org.apache.accumulo.core.master.thrift.MasterClientService$Client.recv_waitForTableOperation(MasterClientService.java:684)
        at org.apache.accumulo.core.master.thrift.MasterClientService$Client.waitForTableOperation(MasterClientService.java:665)
  .
  .
  .

Looked in the master log around that time and saw the following :

05 21:11:02,560 [fate.Fate] WARN : Failed to execute Repo, tid=07a6674a2cad866e
java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server xxx.xxx.xxx.xxx:9997
        at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.hasNext(TabletServerBatchReaderIterator.java:188)
        at org.apache.accumulo.server.master.state.MetaDataTableScanner.hasNext(MetaDataTableScanner.java:93)
        at org.apache.accumulo.server.master.tableOps.CleanUp.isReady(DeleteTable.java:94)

Went to that tablet server and saw the following :

05 21:11:02,523 [tabletserver.TabletServer] WARN : exception while doing multi-scan
java.lang.IllegalStateException: never been seeked
        at org.apache.accumulo.core.iterators.WrappingIterator.hasTop(WrappingIterator.java:71)
        at org.apache.accumulo.core.iterators.user.VersioningIterator.hasTop(VersioningIterator.java:81)
        at org.apache.accumulo.core.iterators.user.WholeRowIterator.hasTop(WholeRowIterator.java:207)
        at org.apache.accumulo.server.master.state.TabletStateChangeIterator.consume(TabletStateChangeIterator.java:113)
        at org.apache.accumulo.core.iterators.SkippingIterator.seek(SkippingIterator.java:38)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:118)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:105)
        at org.apache.accumulo.server.tabletserver.Tablet.lookup(Tablet.java:1622)
        at org.apache.accumulo.server.tabletserver.Tablet.lookup(Tablet.java:1706)

Looking at the code I think the WholeRowIterator may call hasTop on its source iterator w/o seeking it sometimes. This is because its seek function may return w/o seeking the source iterator.