[SYSTEMDS-3147] Empty part files, when writing text

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemDS 2.2
• Assignee: Matthias Boehm
• Reporter: Sebastian Baunsgaard
• Created: 30/Sep/21 18:27
• Updated: 12/Oct/21 21:15
• Resolved: 12/Oct/21 21:15

Description

Script:

X = read($1)
write(X, "test/amazon.txt", format="text")

call:

systemds readA.dml -debug -config noConf.xml -args "../experiments/data/amazon/train_amazon.data"

Full path for dataset:

/home/sbaunsgaard/github/papers/2021-vldb-LCLA/experiments/data/amazon/train_amazon.data

environment:

SYSTEMDS_STANDALONE_OPTS=-Xmx110g -Xms110g -Xmn11g
SYSTEMDS_DISTRIBUTED_OPTS=        --master yarn         --deploy-mode client         --driver-memory 100g         --conf spark.driver.extraJavaOptions="-Xms100g -Xmn10g -Dlog4j.configuration=file:code/conf/log4j-factory.properties"         --conf spark.executor.extraJavaOptions="-Dlog4j.configuration=file:code/conf/log4j-factory.properties"         --conf spark.executor.heartbeatInterval=100s         --files code/conf/log4j-factory.properties         --conf spark.network.timeout=512s         --num-executors 10         --executor-memory 105g         --executor-cores 32         
SYSTEMDS_ROOT=/home/sbaunsgaard/github/systemds

leads to only one of the part files containing data.

CHARLIE:~/github/papers/2021-vldb-LCLA/comp3Paper$ ll -h test/amazon.txt/
total 358M
drwxrwxrwx 2 sbaunsgaard cluster 4.0K Oct 12 15:04 ./
drwxr-xr-x 5 sbaunsgaard cluster 4.0K Oct 12 15:04 ../
-rw-r--r-- 1 sbaunsgaard cluster 358M Oct 12 15:04 0-m-00000
-rw-r--r-- 1 sbaunsgaard cluster    0 Oct 12 15:04 0-m-00001
-rw-r--r-- 1 sbaunsgaard cluster    0 Oct 12 15:04 0-m-00002
-rw-r--r-- 1 sbaunsgaard cluster    0 Oct 12 15:04 0-m-00003
-rw-r--r-- 1 sbaunsgaard cluster    0 Oct 12 15:04 0-m-00004
-rw-r--r-- 1 sbaunsgaard cluster    0 Oct 12 15:04 0-m-00005