[ACCUMULO-3745] deadlock in SourceSwitchingIterator

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.5.0, 1.5.1, 1.5.2, 1.6.0, 1.6.1, 1.6.2
• Fix Version/s: 1.5.3, 1.6.3, 1.7.0
• Component/s: tserver

Environment

Large production cluster, with complex iterator trees.

• Assignee: Keith Turner Assign to me
• Reporter: Eric C. Newton
• Created: 22/Apr/15 16:11
• Updated: 18/May/15 19:27
• Resolved: 27/Apr/15 22:57

Description

Details come from an offline cluster, so it's difficult to reproduce the exact details. A very complex iterator was running over tablet. "deepCopy" may have been called a couple dozen times, which may have contributed to the problem.

Relevant facts:

A scan and a minor compaction created a deadlock which was detected by the java runtime.

"Query... ":
  waiting to lock monitor 0x1234 (object 0x1234, a java.util.Collections$SynchronizedRandomAccessList), 
  which is held by "minor compactor 1"
"minor compactor 1":
 waiting to lock monitor 0x9876 (object 0x9876, a org.apache.accumulo.core.iterators.system.SourceSwitchingIterator), 
 which is held by "Query..."

Java stacks:

"Query..."
  at [e-mail](Collections.java:1636)
  - waiting to lock <0x1234> (a java.util.Collections$SynchronizedRandomAccessList)
  at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.<init>(SourceSwitchingIterator.java:72)
 at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.deepCopy(SourceSwitchingIterator:85)
 - locked <0x9876> (a org.apache.accumulo.core.iterators.system.SourceSwitchingIterator)
  ... PartialMutationSkippingIterator.deepCopy(InMememoryMap.java:113)
 ... InMemoryMap#MemoryIterator.deepCopy(InnMemoryMap.java:623)
 ...

and:

"minor compactor 1":
 at org.apache.accumulo.core.iterators.system.SourceSwitchingIterarot._switchNow(SourceSwitchingIterator:171)
 - waiting to lock <0x9876> (a org.apache.accumulo.core.iterators.system.SourceSwitchingIterator)
 at org.apache.accumulo.iterators.system.SourceSwitchingIterator.switchNow(SourceSwitchingIterator.java:184)
 locked <0x1234> (a java.util.Collections#SynhronizedRandomAccessList)
 at org.apache.accumulo.tserver.InMemoryMap$MemoryIterator.switchNow(InMemoryMap.java:647)
 ...