[SYSTEMDS-1310] Parfor block partitioning (mini batches)

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: SystemML 0.14
• Component/s: APIs, Runtime
• Assignee: Matthias Boehm
• Reporter: Matthias Boehm
• Created: 19/Feb/17 07:20
• Updated: 21/Apr/17 05:02
• Resolved: 21/Mar/17 19:40

Description

This task aims to add basic support for block partitioning of fixed size batches of rows or columns in the parfor optimizer and runtime. For now, we will only apply this for the spark execution backend. The basic supported form looks as follows, where we require that batches are aligned with the systemml blocksize.

N = 20;
parfor(bi in 1 : nrow(X)/N) {
   Xbi = X[((bi-1)*N+1):(bi*N), ];
   R[bi, 1] = sum(Xbi);
}

Issue Links

• is duplicated by: SYSTEMDS-230 Blockwise data partitioning
• relates to: SYSTEMDS-1159 Enable Remote Hyperparameter Tuning