[SINGA-37] Enable users to set parameter sharing in model configuration

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Unassigned
• Reporter: wangwei
• Created: 19/Jul/15 02:33
• Updated: 25/Jul/15 06:07
• Resolved: 25/Jul/15 06:07

Description

For some models (e.g., RBM, RNN), the parameters of some layers are shared (tied). To enable this feature, SINGA needs to provide a configuration field which indicates from whom to share the parameter.

In specific, we can add one field in the ParamProto:

// the name of a Param who owns the Param and shares its values with this Param.
optional string share_from;  

The NeuralNet class with share the Params based on the share_from field. Params only share values, i.e., data field. Everyone has its own gradient, history, etc fields. SINGA will automatically average the gradients for the shared Params before sending the update request.

This ticket will not consider the case in which the original (owner of a shared) Param object is partitioned due to model partitioning, e.g., the weight matrix is partitioned into two sub-matrix if the inner-product layer is partitioned on dimension 1.