Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
The error happens when in hybrid mode:
first generating a binary file. with a config on non standard block size
# X = rand(rows = 8026324, cols = 2330066, min = 0, max = 5, sparsity = 0.000001203) # write(X, "test/generated.data", format="binary")
Then reading it and directly writing it in other format with standard config: (note that if you process the value in any way it works.)
X = read("test/generated.data") # X = X / 2 write(X, "test/generated.txt", format="text")
An Error Occured :
DMLRuntimeException -- org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 4 and 6 -- Error evaluating instruction: CP°write°pREADX·MATRIX·FP64°test/generated.txt·SCALAR·STRING·true°text·SCALAR·STRING·true°·SCALAR·STRING·true°-1
DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 4 and 6 -- Error evaluating instruction: CP°write°pREADX·MATRIX·FP64°test/generated.txt·SCALAR·STRING·true°text·SCALAR·STRING·true°·SCALAR·STRING·true°-1
DMLRuntimeException -- Reading of test/generated.data (1530295320) failed.
IOException -- Failed parallel read of binary block input.
ExecutionException -- java.lang.ArrayIndexOutOfBoundsException
ArrayIndexOutOfBoundsException -- null
ArrayIndexOutOfBoundsException -- 7
org.apache.sysds.runtime.DMLRuntimeException: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 4 and 6 -- Error evaluating instruction: CP°write°pREADX·MATRIX·FP64°test/generated.txt·SCALAR·STRING·true°text·SCALAR·STRING·true°·SCALAR·STRING·true°-1
at org.apache.sysds.runtime.controlprogram.Program.execute(Program.java:160)
at org.apache.sysds.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:89)
at org.apache.sysds.api.DMLScript.execute(DMLScript.java:431)
at org.apache.sysds.api.DMLScript.executeScript(DMLScript.java:274)
at org.apache.sysds.api.DMLScript.main(DMLScript.java:169)
Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 4 and 6 -- Error evaluating instruction: CP°write°pREADX·MATRIX·FP64°test/generated.txt·SCALAR·STRING·true°text·SCALAR·STRING·true°·SCALAR·STRING·true°-1
at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:290)
at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:203)
at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125)
at org.apache.sysds.runtime.controlprogram.Program.execute(Program.java:154)
... 4 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: Reading of test/generated.data (1530295320) failed.
at org.apache.sysds.runtime.controlprogram.caching.CacheableData.exportData(CacheableData.java:882)
at org.apache.sysds.runtime.controlprogram.caching.CacheableData.exportData(CacheableData.java:803)
at org.apache.sysds.runtime.controlprogram.caching.CacheableData.exportData(CacheableData.java:799)
at org.apache.sysds.runtime.instructions.cp.VariableCPInstruction.processWriteInstruction(VariableCPInstruction.java:1010)
at org.apache.sysds.runtime.instructions.cp.VariableCPInstruction.processInstruction(VariableCPInstruction.java:636)
at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:254)
... 7 more
Caused by: java.io.IOException: Failed parallel read of binary block input.
at org.apache.sysds.runtime.io.ReaderBinaryBlockParallel.readBinaryBlockMatrixFromHDFS(ReaderBinaryBlockParallel.java:121)
at org.apache.sysds.runtime.io.ReaderBinaryBlockParallel.readMatrixFromHDFS(ReaderBinaryBlockParallel.java:76)
at org.apache.sysds.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:235)
at org.apache.sysds.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:188)
at org.apache.sysds.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS(MatrixObject.java:438)
at org.apache.sysds.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS(MatrixObject.java:66)
at org.apache.sysds.runtime.controlprogram.caching.CacheableData.readBlobFromHDFS(CacheableData.java:1052)
at org.apache.sysds.runtime.controlprogram.caching.CacheableData.exportData(CacheableData.java:873)
... 12 more
Caused by: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException
at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006)
at org.apache.sysds.runtime.io.ReaderBinaryBlockParallel.readBinaryBlockMatrixFromHDFS(ReaderBinaryBlockParallel.java:111)
... 19 more
Caused by: java.lang.ArrayIndexOutOfBoundsException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
... 20 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 7
at org.apache.sysds.runtime.data.SparseRowVector.append(SparseRowVector.java:236)
at org.apache.sysds.runtime.data.SparseBlockMCSR.append(SparseBlockMCSR.java:333)
at org.apache.sysds.runtime.matrix.data.MatrixBlock.appendRowToSparse(MatrixBlock.java:830)
at org.apache.sysds.runtime.matrix.data.MatrixBlock.appendToSparse(MatrixBlock.java:802)
at org.apache.sysds.runtime.matrix.data.MatrixBlock.appendToSparse(MatrixBlock.java:788)
at org.apache.sysds.runtime.io.ReaderBinaryBlockParallel$ReadFileTask.call(ReaderBinaryBlockParallel.java:190)
at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)