[IGNITE-21924] Improve user threads handling

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-beta2
• Labels: ignite-3 storage-threading threading
• Assignee: Roman Puchkovskiy
• Reporter: Roman Puchkovskiy
• Reviewer: Aleksandr Polovtsev
• Created: 03/Apr/24 07:38
• Updated: 09/Apr/24 07:06
• Resolved: 09/Apr/24 07:06

Description

Now, some Ignite threads are marked with ThreadAttributes, but some are not. ThreadAttributes contain the set of operations (like storage read, storage write, blocking on a lock, etc) a thread is allowed to do. If an attribute marked with these attributes tries to execute an operation, we know what to do: if it's not allowed to, we throw a thread assertion (if they are enabled).

Also, in ReplicaManager there is a stage where we either let the thread to proceed execution (and potentially touch a storage), or we resubmit it to a thread that is guaranteed to be allowed to touch storages. We need to make a decision on whether we need to resubmit or not.

Threads that do not implement ThreadAttributes are:

1. Ignite threads which were not 'marked'
2. User threads
3. JRE threads (like the common fork join pool threads and delayerthread) to which the execution might be switched accidentally (due to a bug in Ignite)

If a storage operation is executed in a non-marked Ignite thread or in a JRE thread, this is an indication of a bug. But for user threads, currently either allow anything, or treat them as 'they are not allowed anything', having to use the ugly technique of whitelisting the threads.

This has to be sorted out.

With the user threads, the situation should be treated in a more complex way:

• If the user entered our public API via a sync operation, such a thread must be allowed to do anything
• If the entry point is an async operation, such a thread should NOT be allowed to touch the storage (as a storage operation might take block on taking locks, and it's not a correct thing to block a user thread if the operation is async)

In this way, we will always know what to do with a thread trying to touch the storage:

1. If it is marked with ThreadAttributes, consult the allowed operations, and if the operation is not allowed, throw an assertion
2. If it's not marked, but its thread-local SyncOperation flag is raised, it's allowed anything
3. Otherwise, we throw an assertion (as it's either a not-marked Ignite thread [should be marked], or it's a JRE thread [we should not allow it to touch the storages], or it's a user's thread which executes an async operation [should not touch the storages])

Also, the criterion for resubmitting in ReplicaManager is: (is ThreadAttributes and there is no permission to touch storages) OR (!ThreadAttributes AND it's a user's AsyncOperation thread). This will result in resubmitting for getAsync (and its varieties) and for the cases when we come from a Metastorage thread.

Issue Links

• links to: GitHub Pull Request #3547