[SYSTEMDS-2855] Federated LM Pipeline transfers X to coordinator

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemDS 2.1
• Assignee: Matthias Boehm
• Reporter: Matthias Boehm
• Created: 09/Feb/21 15:09
• Updated: 10/Feb/21 12:48
• Resolved: 10/Feb/21 12:48

Description

Due to a complex series of events, the federated LM pipeline test pulls the entire federated feature matrix into the coordinator. This happens for a single operation before entering the lmCG main loop: r = - t(X) %*% y;.

In detail, the hop-lop transpose-mm rewrite does not apply because the minus is not pulled after the matrix multiplication. This happens because lmCG is subject to recompilation on function entry which does not apply simplification rewrites. Then we transpose X from row to column partitioned federated matrix, and this operation is not yet supported.

The fix is relatively simple: apply simplification rewrites during function recompile (good for many scripts), and add the missing support for federated matrix multiplication of column partitioned federated left-hand-sides.