[SYSTEMDS-2476] Unexpected mapreduce task

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Fix Version/s: SystemDS 2.0
• Epic Link: Language and runtime for parameter servers
• Assignee: Matthias Boehm
• Reporter: LI Guobao
• Created: 31/Jul/18 20:22
• Updated: 19/May/20 19:46
• Resolved: 19/May/20 19:46

Description

When trying to use scalar casting to get element from a list, unexpected mapreduce tasks are launched instead of CP mode. The scenario is to replace C = 1 with C = as.scalar(hyperparams["C"]) inside the gradient function found in src/test/scripts/functions/paramserv/mnist_lenet_paramserv.dml. And then the problem could be reproduced by launching the method testParamservBSPBatchDisjointContiguous inside class org.apache.sysml.test.integration.functions.paramserv.ParamservLocalNNTest

Here is the stack:

18/07/31 22:10:27 INFO mapred.MapTask: numReduceTasks: 1
18/07/31 22:10:27 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
18/07/31 22:10:27 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
18/07/31 22:10:27 INFO mapred.MapTask: soft limit at 83886080
18/07/31 22:10:27 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
18/07/31 22:10:27 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
18/07/31 22:10:27 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
18/07/31 22:10:27 INFO mapreduce.Job: Running job: job_local792652629_0008

mboehm7, if possible, could you take a look on this? And I've double checked the creation of execution context in ParamservBuiltinCPInstruction. But it is instance of ExecutionContext not SparkExecutionContext.