[SYSTEMDS-2950] Threads wait forever for a removed entry in the lineage cache

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemDS 2.1
• Assignee: Arnab Phani
• Reporter: Arnab Phani
• Created: 26/Apr/21 12:34
• Updated: 26/Apr/21 14:06
• Resolved: 26/Apr/21 14:06

Description

With parfor, we ensure reuse across threads. The first thread puts a placeholder in the cache and continues executing, where the other threads wait on that placeholder. At the end of the execution, the first thread fills the placeholder with data and wakes up all the other threads, so that they can reuse and skip executing the instruction.

However, in some cases (e.g. output is a frame, output is federated, output size is more than cache size, etc.) the first task simply removes the placeholder after executing the instruction, which means all the other sleeping threads wait forever. This leads to a hung state.