[SYSTEMDS-1349] Parfor optimizer support for shared reads (lower memory req)

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemML 1.2
• Component/s: Compiler
• Assignee: Matthias Boehm
• Reporter: Matthias Boehm
• Created: 24/Feb/17 01:01
• Updated: 20/Apr/18 08:46
• Resolved: 20/Apr/18 08:46

Description

Let's assume the following example script and focus on local parfor with a memory budget of 10GB and k=16 threads (but this issue similarly applies to parfor spark).

X = rand(rows=1000000, cols=1000); #8GB
parfor(i in 1:ncol(X)) 
    print(sum(X[,i]));

The memory estimate of the right indexing operation is 8GB and the parfor optimizer tries to set k=min(floor(10GB/8GB), 16) and hence falls back to a single-threaded plan with k=1. However, this is completely unnecessary because X is a shared-read (i.e., variable that is never updated in the parfor body). Accordingly, we should determine the degree of parallelism as k=min(floor((10GB-8GB)/8MB), 16).

The challenge is to do this for arbitrary control flow and multiple shared reads of different dimension and sparsity.

Issue Links

• is depended upon by: SYSTEMDS-1313 Parfor broadcast exploitation
• is related to: SYSTEMDS-2251 Improve parfor optimizer set degree of parallelism
• relates to: SYSTEMDS-1879 Parfor remote spark w/ reuse of shared inputs