Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-3745

deadlock in SourceSwitchingIterator

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.5.0, 1.5.1, 1.5.2, 1.6.0, 1.6.1, 1.6.2
    • Fix Version/s: 1.5.3, 1.6.3, 1.7.0
    • Component/s: tserver
    • Labels:
      None
    • Environment:

      Large production cluster, with complex iterator trees.

      Description

      Details come from an offline cluster, so it's difficult to reproduce the exact details. A very complex iterator was running over tablet. "deepCopy" may have been called a couple dozen times, which may have contributed to the problem.

      Relevant facts:

      A scan and a minor compaction created a deadlock which was detected by the java runtime.

      "Query... ":
        waiting to lock monitor 0x1234 (object 0x1234, a java.util.Collections$SynchronizedRandomAccessList), 
        which is held by "minor compactor 1"
      "minor compactor 1":
       waiting to lock monitor 0x9876 (object 0x9876, a org.apache.accumulo.core.iterators.system.SourceSwitchingIterator), 
       which is held by "Query..."
      

      Java stacks:

      "Query..."
        at java.util.Collections@SynchronizedCollection.add(Collections.java:1636)
        - waiting to lock <0x1234> (a java.util.Collections$SynchronizedRandomAccessList)
        at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.<init>(SourceSwitchingIterator.java:72)
       at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.deepCopy(SourceSwitchingIterator:85)
       - locked <0x9876> (a org.apache.accumulo.core.iterators.system.SourceSwitchingIterator)
        ... PartialMutationSkippingIterator.deepCopy(InMememoryMap.java:113)
       ... InMemoryMap#MemoryIterator.deepCopy(InnMemoryMap.java:623)
       ...
      

      and:

      "minor compactor 1":
       at org.apache.accumulo.core.iterators.system.SourceSwitchingIterarot._switchNow(SourceSwitchingIterator:171)
       - waiting to lock <0x9876> (a org.apache.accumulo.core.iterators.system.SourceSwitchingIterator)
       at org.apache.accumulo.iterators.system.SourceSwitchingIterator.switchNow(SourceSwitchingIterator.java:184)
       locked <0x1234> (a java.util.Collections#SynhronizedRandomAccessList)
       at org.apache.accumulo.tserver.InMemoryMap$MemoryIterator.switchNow(InMemoryMap.java:647)
       ...
      

        Attachments

        1. ACCUMULO-3745-1.patch
          3 kB
          Keith Turner
        2. ACCUMULO-3745-2.patch
          5 kB
          Keith Turner
        3. ACCUMULO-3745-3.patch
          6 kB
          Keith Turner

          Activity

            People

            • Assignee:
              kturner Keith Turner
              Reporter:
              ecn Eric C. Newton
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 20m
                1h 20m