Concurrency problem when doing load tests

• Key: DIRSERVER-624
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Blocker
• Assignee: Alex Karasulu
• Reporter: Emmanuel Lecharny
• Affects Version/s: 1.0-RC3
• Fix Version/s: 1.5.0, 1.0-RC4

Description

When trying to do some load tests (adding 100K users, deleting them, adding them again ...) on a bi-processor computer, we get some concurrency access errors on the secong add/del run :
           _ _ ____ ____ / \ _ __ __ _ ___| |__ ___| _ \/ ___|
         / _ \ | '_ \ / _` |/ __| '_ \ / _ \ | | \___ \ / ___ \| |_) | (_| | (__| | | | __/ |_| |___) |
       /_/ \_\ .__/ \__,_|\___|_| |_|\___|____/|____/ |_|
 [08:00:07] WARN [org.apache.directory.server.core.DefaultDirectoryService] - You didn't change the admin password of directory service instance 'default'. Please update the admin password as soon as possible to prevent a possible security breach.
[10:33:24] WARN [org.apache.directory.server.ldap.LdapProtocolProvider$LdapProtocolHandler] - [/193.140.236.75:42091] Unexpected exception forcing session to close: sending disconnect notice to client. java.util.ConcurrentModificationException
        at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1031) at java.util.TreeMap$KeyIterator.next(TreeMap.java:1058)
        at org.apache.directory.server.core.partition.impl.btree.TupleEnumeration.next(TupleEnumeration.java:66)
        at org.apache.directory.server.core.partition.impl.btree.IndexEnumeration.prefetch(IndexEnumeration.java:151)
        at org.apache.directory.server.core.partition.impl.btree.IndexEnumeration.next(IndexEnumeration.java:94)
        at org.apache.directory.server.core.partition.impl.btree.IndexAssertionEnumeration.prefetch(IndexAssertionEnumeration.java:155)
        at org.apache.directory.server.core.partition.impl.btree.IndexAssertionEnumeration.next(IndexAssertionEnumeration.java:117)
        at org.apache.directory.server.core.partition.impl.btree.DisjunctionEnumeration.next(DisjunctionEnumeration.java:163)
        at org.apache.directory.server.core.partition.impl.btree.IndexAssertionEnumeration.prefetch(IndexAssertionEnumeration.java:155)
        at org.apache.directory.server.core.partition.impl.btree.IndexAssertionEnumeration.next(IndexAssertionEnumeration.java:117)
        at org.apache.directory.server.core.partition.impl.btree.BTreeSearchResultEnumeration.next(BTreeSearchResultEnumeration.java:99)
        at org.apache.directory.server.core.enumeration.SearchResultFilteringEnumeration.prefetch(SearchResultFilteringEnumeration.java:287)
        at org.apache.directory.server.core.enumeration.SearchResultFilteringEnumeration.next(SearchResultFilteringEnumeration.java:201)
        at org.apache.directory.server.core.enumeration.SearchResultFilteringEnumeration.prefetch(SearchResultFilteringEnumeration.java:287)
        at org.apache.directory.server.core.enumeration.SearchResultFilteringEnumeration.next(SearchResultFilteringEnumeration.java:201)
        at org.apache.directory.server.core.enumeration.SearchResultFilteringEnumeration.prefetch(SearchResultFilteringEnumeration.java:287)
        at org.apache.directory.server.core.enumeration.SearchResultFilteringEnumeration.next(SearchResultFilteringEnumeration.java:201)
        at org.apache.directory.server.ldap.support.SearchResponseIterator.next(SearchResponseIterator.java:185)
        at org.apache.directory.server.ldap.support.SearchHandler.messageReceived(SearchHandler.java:281)
        at org.apache.mina.handler.demux.DemuxingIoHandler.messageReceived(DemuxingIoHandler.java:128)
        at org.apache.directory.server.ldap.LdapProtocolProvider$LdapProtocolHandler.messageReceived(LdapProtocolProvider.java:431)
        at org.apache.mina.common.support.AbstractIoFilterChain$2.messageReceived(AbstractIoFilterChain.java:188)
        at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:501)
        at org.apache.mina.common.support.AbstractIoFilterChain.access$1000(AbstractIoFilterChain.java:51)
        at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:787)
        at org.apache.mina.filter.LoggingFilter.messageReceived(LoggingFilter.java:95)
        at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:501)
        at org.apache.mina.common.support.AbstractIoFilterChain.access$1000(AbstractIoFilterChain.java:51)
        at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:787)
        at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:187)
        at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:501)
        at org.apache.mina.common.support.AbstractIoFilterChain.access$1000(AbstractIoFilterChain.java:51)
        at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:787)
        at org.apache.mina.filter.ThreadPoolFilter.processEvent(ThreadPoolFilter.java:718)
        at org.apache.mina.filter.ThreadPoolFilter$Worker.processEvents(ThreadPoolFilter.java:474)
        at org.apache.mina.filter.ThreadPoolFilter$Worker.run(ThreadPoolFilter.java:429)

Comments

This problem occurs when trying to search for an entry while loading some

[Simon Temple]

This problem appears more fundamental.

I have one thread in the same VM as a DS and one thread running in my eclipse - both running searches across the same area of the directory.

Very soon one or other thread hits this concurrency problem during the enumeration of results.

We started to look at BTreeDirectoryPartition as it 'appears' to only create a single DefaultSearchEngine and ExpressionEnumerator... but pretty soon started to get lost in the complexity.

If anyone can provide us with more of an idea where we should start, we'll gladly investigate further.

TIA

[Alex Karasulu]

Fixed here for 1.0 branch:

    http://svn.apache.org/viewvc?view=rev&revision=437489

and for 1.1 branch here:

    http://svn.apache.org/viewvc?view=rev&revision=437490