[SYSTEMDS-1884] Unnecessary spark/mr instructions on removeEmpty select

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: SystemML 0.14
• Fix Version/s: SystemML 0.15
• Assignee: Matthias Boehm
• Reporter: Matthias Boehm
• Created: 03/Sep/17 01:47
• Updated: 08/Sep/17 19:31
• Resolved: 08/Sep/17 19:31

Description

The builtin function removeEmpty takes an optional select parameter. If this parameter is provided, we directly use it instead of explicitly computing it. However, the compilation of removeEmpty handled this incorrectly, setting the forced execution type to the initially compiled execution type of removeEmpty. This is problematic in scenarios with dynamic recompilation. For example, consider the following scenario:

C = replace(target=A, pattern=2, replacement=0);
D = removeEmpty(B, margin="rows", select=C);

If the removeEmpty is initially compiled to SPARK, this also mistakenly set the forced execution type of replace to SPARK. During recompilation, the removeEmpty might be compiled to CP but because replace has a forced execution type, it will remain in SPARK.

Workaround: The workaround for this issue is to put a while(FALSE) just before the removeEmpty call (more precisely, C in above example needs to be computed in a separate HOP DAG).