[SYSTEMDS-2984] Nested lists remove elements from the dag

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemDS 2.1
• Assignee: Matthias Boehm
• Reporter: Sebastian Baunsgaard
• Created: 28/May/21 11:22
• Updated: 03/Jun/21 11:26
• Resolved: 03/Jun/21 11:26

Description

currently the list primitive does not support rapid access to nested elements, and in many cases the explain show that in simple examples list elements are simply removed from the dag.

Not working:

a_1 = matrix(3,  1, 1)
a_2 = matrix(2,  1,2)
a_3 = matrix(45,  1,3)
b = list(a_1,a_2)
c = list(a_3, b)
print(c)

works:

a_1 = matrix(3, 1, 1)
a_2 = matrix(2, 1,2)
a_3 = matrix(45, 1,3)
b = list(a_1,a_2)
c = list(a_3, b)
print(toString(c))
print(toString(b))
print(toString(a_1))
print(toString(a_2))
print(toString(a_3))

{java}

The explain show why the first does not work because _mVar8 and 9 is removed before they are used inside the list:

1. EXPLAIN (RUNTIME):
2. Memory Budget local/remote = 2832MB/?MB/?MB
3. Degree of Parallelism (vcores) local/remote = 16/
   PROGRAM ( size CP/MR = 14/0 )
   --MAIN PROGRAM
   ----GENERIC (lines 14-19) [recompile=false]
   ------CP createvar _mVar7 scratch_space//_p16187_127.0.1.1//_t0/temp3 true MATRIX binary 1 3 1000 3 copy
   ------CP rand 1.SCALAR.INT64.true 3.SCALAR.INT64.true 1000 45 45 1.0 -1 uniform 1.0 16 _mVar7.MATRIX.FP64
   ------CP createvar _mVar8 scratch_space//_p16187_127.0.1.1//_t0/temp4 true MATRIX binary 1 1 1000 1 copy
   ------CP rand 1.SCALAR.INT64.true 1.SCALAR.INT64.true 1000 3 3 1.0 -1 uniform 1.0 16 _mVar8.MATRIX.FP64
   ------CP createvar _mVar9 scratch_space//_p16187_127.0.1.1//_t0/temp5 true MATRIX binary 1 2 1000 2 copy
   ------CP rand 1.SCALAR.INT64.true 2.SCALAR.INT64.true 1000 2 2 1.0 -1 uniform 1.0 16 _mVar9.MATRIX.FP64
   ------CP list _mVar8.MATRIX.FP64 _mVar9.MATRIX.FP64 _Var10.LIST.UNKNOWN
   ------CP rmvar _mVar8 _mVar9
   ------CP list _mVar7.MATRIX.FP64 _Var10.LIST.UNKNOWN.false _Var11.LIST.UNKNOWN
   ------CP rmvar _mVar7 _Var10
   ------CP toString target=_Var11 _Var12.SCALAR.STRING
   ------CP rmvar _Var11
   ------CP print _Var12.SCALAR.STRING.false _Var13.SCALAR.STRING
   ------CP rmvar _Var12 _Var13

SystemDS Statistics:
Total execution time: 0.019 sec.