[SYSTEMDS-2927] SPOOF memory estimate for exec = singlenode

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: SystemDS 2.1
• Fix Version/s: SystemDS 2.1
• Component/s: Compiler
• Assignee: Unassigned
• Reporter: Mark Dokter
• Created: 12/Apr/21 11:29
• Updated: 24/May/21 20:49
• Resolved: 24/May/21 20:49

Description

When executing SystemDS with -exec singlenode, the blockSize attribute of the SpoofFusedOp HOP is set to -1. This triggers a worst case memory estimate by going into the first conditional block in [1].

The blockSize setting might be intentional in this scenario, but the effect it has on spoof single node operation is unexpected imho, so I'd file this under "maybe a bug".

[1]

org.apache.sysds.hops.OptimizerUtils#estimatePartitionedSizeExactSparsity(long, long, long, double, boolean)