[SYSTEMDS-2400] Dynamic recompilation loses max thread constraints of hops

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemML 1.2
• Assignee: Matthias Boehm
• Reporter: Matthias Boehm
• Created: 16/Jun/18 01:38
• Updated: 16/Jun/18 02:01
• Resolved: 16/Jun/18 02:01

Description

Dynamic recompilation deep copies the hop DAG and applies - among other things - various rewrites. In case these rewrites create new hops (no inplace modification) these rewrites lose the max thread constraints set by parfor and paramserv. On nodes with large degree of parallelism this constitutes a serious issue because it leads to large over-provisioning and thus reduced performance and potential OOMs.

Issue Links

• relates to: SYSTEMDS-2397 Paramserv ASP failing w/ OOM (too many threads)