Uploaded image for project: 'SystemDS'
  1. SystemDS
  2. SYSTEMDS-1567

Remove conditionals from nn layers

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: SystemML 1.0.0
    • Fix Version/s: SystemML 0.15
    • Component/s: APIs
    • Labels:
      None

      Description

      Conditionals in nn layers introduce transient read/write variables that disables fused operators such as CP relu_maxpooling_backward and hence redundant execute sparsity-introducing sel+ operator. This operator causes unnecessary dense-to-sparse-to-dense conversion and becomes the heavy hitter after native BLAS change. Note: some fused operators such as CP relu_maxpooling are still applied because there is no conditional in between those layers.

      Without conditionals in dropout layer: https://github.com/apache/incubator-systemml/blob/master/scripts/nn/layers/dropout.dml#L49-L53

      Iter:2000.0, training loss:0.003149394810197065, training accuracy:100.0
      Iter:2000.0, validation loss:191.9888157354513, validation accuracy:96.875
      SystemML Statistics:
      Total elapsed time:             416.609 sec.
      Total compilation time:         0.000 sec.
      Total execution time:           416.609 sec.
      Number of compiled Spark inst:  69.
      Number of executed Spark inst:  2.
      Native mkl calls (LibMatrixMult/LibMatrixDNN):  4270/10553.
      Cache hits (Mem, WB, FS, HDFS): 277973/0/0/0.
      Cache writes (WB, FS, HDFS):    143616/0/0.
      Cache times (ACQr/m, RLS, EXP): 0.101/0.080/1.988/0.000 sec.
      HOP DAGs recompiled (PRED, SB): 0/2277.
      HOP DAGs recompile time:        6.146 sec.
      Spark ctx create time (lazy):   0.027 sec.
      Spark trans counts (par,bc,col):0/0/0.
      Spark trans times (par,bc,col): 0.000/0.000/0.000 secs.
      Total JIT compile time:         37.746 sec.
      Total JVM GC count:             3949.
      Total JVM GC time:              56.609 sec.
      Heavy hitter instructions (name, time, count):
      -- 1)   conv2d_bias_add         48.984 sec      4514
      -- 2)   conv2d_backward_filter  47.780 sec      4026
      -- 3)   -*      38.246 sec      16104
      -- 4)   +*      35.902 sec      8052
      -- 5)   +       34.227 sec      30566
      -- 6)   ba+*    30.643 sec      12566
      -- 7)   relu_maxpooling_backward        29.678 sec      4026
      -- 8)   conv2d_backward_data    28.520 sec      2013
      -- 9)   *       26.825 sec      35275
      -- 10)  relu_backward   24.842 sec      6039
      

      With conditional, we add sel+ to the heavy hitter:

      -- 1)   sel+    55.054 sec      6283
      

      [~mwdusenb@us.ibm.com] Since you created the layers, I think you should decide how best to restructure the DML. My recommendation would be to create two layers in case of conditionals.

      Matthias Boehm Berthold Reinwald

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              niketanpansare Niketan Pansare
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: