[SYSTEMDS-678] MLContext parallelization

• Type: Question
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: SystemML 0.10
• Fix Version/s: SystemML 0.10
• Component/s: Algorithms, Parser, Runtime
• Assignee: Unassigned
• Reporter: Johannes Wilke
• Created: 10/May/16 07:28
• Updated: 25/Jun/20 09:05
• Resolved: 25/Jun/20 09:05

Description

I try to execute script in the MLContext. It is executing, but it dont parallel. For smaller scripts, it works fine. But this script doesnt and it is not clear why. I think it is because of the 4 loop levels, but I am not sure.
Is there a documentation what is parallizable and what isnt?
If I change the main while-loop, i wish to parallize, to a parfor loop it works.
Here is the script:

X = read($Xin)
P = read($Pin)
#errorMatrix = matrix(0.0,rows=1,cols=1)
j = 1
sum = 0

while (j <=nrow(X) & sum >= 0){ # this should be parallelized
#parfor(j in 1: nrow(X),check=0){

first = TRUE

windows = matrix(0,rows=1,cols=1)

offsetPreWindowDefinitions = 0
sumWindowLength = 0
mastercount = 0
totalwindowLength = 0
s = 0
for(i in 1: nrow(P)){
if((as.scalar(P[i,1])*as.scalar(P[i,2]))>totalwindowLength)

{ totalwindowLength = (as.scalar(P[i,1])*as.scalar(P[i,2])) }

s = s+1
}

lastWindow = matrix(0,rows=sum(P[,1]),cols=1)

for(i in 1:nrow(P)){# for every Window-Definition

for(k in 1: as.integer(as.scalar(P[i,1]))){# for every pnum

column = matrix(0,rows=as.integer(as.scalar(P[1,4])),cols=1)

for(l in 1: nrow(column)+1){
offsetPreWindowDefinitions = totalwindowLength - (as.scalar(P[i,1])*as.scalar(P[i,2]))
tsindex = ((k-1) * as.scalar(P[i,2])) + l-1 + offsetPreWindowDefinitions
if(l==nrow(column)+1)

{ lastWindow[sumWindowLength+k,1] = X[j,tsindex+1] }

else

{ column[l,1] = X[j,tsindex+1] }

mastercount = mastercount +1
#print(mastercount)

}

if(first)

{ first = FALSE; windows = column }

else

{ windows = cbind(windows,column) }

}

sumWindowLength = sumWindowLength + as.scalar(P[i,1])

}

result = matrix(14.3,rows=as.integer(as.scalar(P[1,4])),cols=1)

for(i in totalwindowLength:as.integer(as.scalar(P[1,4]))+totalwindowLength-1)

{ result[i-totalwindowLength+1,1] = X[j,i+1] s = s+1 }

params = solve(windows,result)
print(j)
predict = matrix(0,rows=1, cols=1)
for(i in 1:nrow(lastWindow))

{ predict[1,1] = predict[1,1] + (params[i,1] * lastWindow[i,1]) s = s+1 }

predictscalar = as.scalar(predict[1,1])
targetscalar = as.scalar(X[j,ncol(X)])

sum = sum + ((targetscalar - predictscalar) * (targetscalar - predictscalar))

j = j+1
#write(lastWindow, "/media/johannes/Data/Seafile/UNI/Beleg/sysml_output/lWOut.csv", format="csv", header=TRUE, sep=",", sparse=TRUE);
#write(windows, "/media/johannes/Data/Seafile/UNI/Beleg/sysml_output/windowsOut.csv", format="csv", header=TRUE, sep=",", sparse=TRUE);
#write(result, "/media/johannes/Data/Seafile/UNI/Beleg/sysml_output/resultOut.csv", format="csv", header=TRUE, sep=",", sparse=TRUE);
}
print(sum/nrow(X))

I hope that you can help me!