[IGNITE-10511] disco-event-worker can be deadlocked by BinaryContext.metadata running is sys striped pool waiting for cache entry lock

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Ignite Flags: Docs Required
• Assignee: Unassigned
• Reporter: Pavel Voronkin
• Created: 03/Dec/18 14:37
• Updated: 10/Dec/18 14:06
• Resolved: 05/Dec/18 09:13

Description

See attached thread dump:

disco-event-worker hangs on removeExplicitNodeLocks() on GridCacheMapEntry which is held by GridDistributedTxRemoteAdapter acquired in GridCacheMapEntry.innerSet().

CacheObjectBinaryProcessorImpl is waiting on metadata message on discovery, which can be processed due to disco-event-worker is stuck.

Possible fix:

    public void onNodeLeft(final ClusterNode node) {
        if (isDone() || !enterBusy())
            return;

        cctx.mvcc().removeExplicitNodeLocks(node.id(), initialVersion());

        try {
            onDiscoveryEvent(new IgniteRunnable() {
                 @Override public void run() {
                     if (isDone() || !enterBusy())
                         return;
                     
                     ...
                 }
             });
        }
        finally {
            ...
        }
    }

As we can see most of the processing is done async in IgniteRunnable() in exchange-worker.

 

We can move 

    cctx.mvcc().removeExplicitNodeLocks(node.id(), initialVersion());

inside this Runnable's body.

Issue Links

• duplicates: IGNITE-9290 Make remove explicit locks async when node left.
• links to: GitHub Pull Request #5563