[SYSTEMDS-2928] CSV parsing with non-default delimiter

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemDS 2.1
• Assignee: Olga Ovcharenko
• Reporter: Olga Ovcharenko
• Created: 12/Apr/21 12:40
• Updated: 13/May/21 21:11
• Resolved: 13/May/21 21:11

Description

After delimiter modification of src/test/scripts/functions/io/csv/in/transfusion_2.single.csv from default delimiter "," to semicolon ";" DML tests failed beacuse reader replaces semicolon with default delimiter and then split and double parsing fail. 

Exception : class java.io.IOException
Message : Read task for csv input failed: java.lang.NumberFormatException: For input string: "2 ,50,12500,98 ,1"
4 > org.apache.sysds.runtime.io.ReaderTextCSVParallel.readCSVMatrixFromHDFS(ReaderTextCSVParallel.java:157)
4 > org.apache.sysds.runtime.io.ReaderTextCSVParallel.readMatrixFromHDFS(ReaderTextCSVParallel.java:102)

To test I modified src/test/scripts/functions/io/csv/ReadCSVTest_2.dml, src/test/scripts/functions/io/csv/csv_verify2.R and src/test/java/org/apache/sysds/test/functions/io/csv/ReadCSVTest.java