[SYSTEMDS-1567] Remove conditionals from nn layers

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: SystemML 1.0.0
• Fix Version/s: SystemML 0.15
• Component/s: APIs
• Epic Link: Deep Learning
• Assignee: Unassigned
• Reporter: Niketan Pansare
• Created: 30/Apr/17 17:36
• Updated: 09/Sep/17 05:24
• Resolved: 01/May/17 17:25

Description

Conditionals in nn layers introduce transient read/write variables that disables fused operators such as CP relu_maxpooling_backward and hence redundant execute sparsity-introducing sel+ operator. This operator causes unnecessary dense-to-sparse-to-dense conversion and becomes the heavy hitter after native BLAS change. Note: some fused operators such as CP relu_maxpooling are still applied because there is no conditional in between those layers.

Without conditionals in dropout layer: https://github.com/apache/incubator-systemml/blob/master/scripts/nn/layers/dropout.dml#L49-L53

Iter:2000.0, training loss:0.003149394810197065, training accuracy:100.0
Iter:2000.0, validation loss:191.9888157354513, validation accuracy:96.875
SystemML Statistics:
Total elapsed time:             416.609 sec.
Total compilation time:         0.000 sec.
Total execution time:           416.609 sec.
Number of compiled Spark inst:  69.
Number of executed Spark inst:  2.
Native mkl calls (LibMatrixMult/LibMatrixDNN):  4270/10553.
Cache hits (Mem, WB, FS, HDFS): 277973/0/0/0.
Cache writes (WB, FS, HDFS):    143616/0/0.
Cache times (ACQr/m, RLS, EXP): 0.101/0.080/1.988/0.000 sec.
HOP DAGs recompiled (PRED, SB): 0/2277.
HOP DAGs recompile time:        6.146 sec.
Spark ctx create time (lazy):   0.027 sec.
Spark trans counts (par,bc,col):0/0/0.
Spark trans times (par,bc,col): 0.000/0.000/0.000 secs.
Total JIT compile time:         37.746 sec.
Total JVM GC count:             3949.
Total JVM GC time:              56.609 sec.
Heavy hitter instructions (name, time, count):
-- 1)   conv2d_bias_add         48.984 sec      4514
-- 2)   conv2d_backward_filter  47.780 sec      4026
-- 3)   -*      38.246 sec      16104
-- 4)   +*      35.902 sec      8052
-- 5)   +       34.227 sec      30566
-- 6)   ba+*    30.643 sec      12566
-- 7)   relu_maxpooling_backward        29.678 sec      4026
-- 8)   conv2d_backward_data    28.520 sec      2013
-- 9)   *       26.825 sec      35275
-- 10)  relu_backward   24.842 sec      6039

With conditional, we add sel+ to the heavy hitter:

-- 1)   sel+    55.054 sec      6283

[e-mail] Since you created the layers, I think you should decide how best to restructure the DML. My recommendation would be to create two layers in case of conditionals.

mboehm7 reinwald