[SYSTEMDS-618] Deep Learning DML Library

• Type: Epic
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: Algorithms
• Epic Name: Deep Learning DML Library
• Assignee: Mike Dusenberry
• Reporter: Mike Dusenberry
• Created: 06/Apr/16 20:23
• Updated: 17/Oct/17 01:04

Description

This issue tracks the creation of a layers-based deep learning library in pure DML.

The library contains layers with simple forward (function evaluation) and backward (gradient computation) functions for affine, convolution (start with 2D), max-pooling, non-linearities (relu, sigmoid, softmax, etc.), dropout, loss functions, other layers, optimizers, and gradient checks.

Examples: Please see example scripts and notebooks in the examples folder: https://github.com/apache/systemml/tree/master/scripts/nn/examples.

SystemML-NN: https://github.com/apache/systemml/tree/master/scripts/nn

• Layers:
  • Core:
    • Affine
    • Batch Normalization 1D
    • Batch Normalization 2D ("Spatial Batch Normalization")
    • Convolution 2D ("Spatial Convolution")
    • LSTM
    • Max Pooling 2D ("Spatial Max Pooling")
    • RNN
  • Nonlinearities:
    • ReLU
    • Sigmoid
    • Softmax
    • Softmax 2D
    • Tanh
  • Loss:
    • Cross-entropy loss
    • Cross-entropy loss 2D
    • L1 loss
    • L2 loss
    • Log ("Logistic") loss
  • Regularization:
    • Dropout
    • L1 reg
    • L2 reg
• Optimizers:
  • Adagrad
  • Adam
  • RMSprop
  • SGD
  • SGD w/ Momentum
  • SGD w/ Nesterov Momentum
• Tests:
  • Gradient Checks
  • Unit Tests

Issue Links

• is depended upon by: SYSTEMDS-1185 SystemML Breast Cancer Project
• is part of: SYSTEMDS-540 Deep Learning
• is related to: SYSTEMDS-409 Extended update in-place support
• is related to: SYSTEMDS-669 Improve PyDML Language
• is related to: SYSTEMDS-1566 Possible regression from 0.13 -> 0.14 for MNIST LeNet script
• is related to: SYSTEMDS-1621 `max(0, X)` fails with type mismatch
• is related to: SYSTEMDS-1686 Transpose Conv2d has incorrect filter shape and incorrect input size argument
• is related to: SYSTEMDS-633 Improve Left-Indexing Performance with (Nested) Parfor Loops in UDFs
• is related to: SYSTEMDS-845 Compare Performance of LeNet Scripts With & Without Using SystemML-NN
• is related to: SYSTEMDS-1561 Improve constant folding during compilation
• is related to: SYSTEMDS-1000 Allow users to pass non-1 bias filler in conv_builtin.dml
• is related to: SYSTEMDS-1479 Make Caffe2DML feature-complete
• relates to: SYSTEMDS-716 Consumability of SystemML for Deep Learning