[SYSTEMDS-1809] Optimize the performance of the distributed MNIST_LeNet_Sgd model training

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: SystemML 1.0.0
• Labels: RemoteParForSpark deeplearning
• Sprint: Sprint 3
• Assignee: Fei Hu
• Reporter: Fei Hu
• Created: 26/Jul/17 17:04
• Updated: 08/Aug/17 17:49

Description

For the current version, the distributed MNIST_LeNet_Sdg model training can be optimized from the following aspects:

1. Optimize the DML scripts with considering the backend engine, such as the intermediate matrixes are exported to HDFS, so we need to avoid unnecessary intermediate matrixes.
2. Improve the efficiency of matrix subsetting
3. Data locality: for RemoteParForSpark, the tasks are parallelized without considering data locality. It will cause a lot of data shuffling if the volume of the input data size is large;
4. Result merge: the current experiments indicate that the result merge part took more time than model training.

After the optimization, we need to compare the performance with the distributed Tensorflow.

Issue Links

• is a parent of: SYSTEMDS-1821 Improve the training process in mnist_lenet_distrib_sgd.dml
• is a parent of: SYSTEMDS-1830 Improve the data locality for the tasks in ParFor body
• is a parent of: SYSTEMDS-1831 Improve the efficiency of matrix subsetting