[IGNITE-12451] Introduce deadlock detection for cache entry reentrant locks

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.6
• Fix Version/s: 2.10
• Labels: important
• Release Note: Fixed java level deadlock on atomic putAll/removeAll operations
• Ignite Flags: Release Notes Required
• Assignee: Aleksey Plekhanov
• Reporter: Ivan Rakov
• Created: 13/Dec/19 15:42
• Updated: 23/Mar/21 15:38
• Resolved: 28/Oct/20 12:17

Description

Aside from IGNITE-12365, we still have possible threat of cache-entry-level deadlock in case of careless usage of JCache mass operations (putAll, removeAll):
1. If two different user threads will perform putAll on the same two keys in reverse order (primary node for which is the same), there's a chance that sys-stripe threads will be deadlocked.
2. Even without direct contract violation from user side, HashMap can be passed as argument for putAll. Even if user threads have called mass operations with two keys in the same order, HashMap iteration order is not strictly defined, which may cause the same deadlock.

Local deadlock detection should mitigate this issue. We can create a wrapper for ReentrantLock with logic that performs cycle detection in wait-for graph in case we are waiting for lock acquisition for too long. Exception will be thrown from one of the threads in such case, failing user operation, but letting the system make progress.

Issue Links

• blocks: IGNITE-6804 Print a warning if HashMap is passed into bulk update operations
• blocks: IGNITE-11558 Developer warning when HashMap is passed to putAll()
• is duplicated by: IGNITE-11657 Deadlock on Cache.putAll(Map<KV>)
• is related to: IGNITE-13653 Don't print warning if unordered map used for bulk update operation on atomic cache
• relates to: IGNITE-11657 Deadlock on Cache.putAll(Map<KV>)
• links to: GitHub Pull Request #8268