Uploaded image for project: 'Singa'
  1. Singa
  2. SINGA-40

Support sparse Param update

Agile BoardAttach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None

    Description

      For some models, e.g., Word2Vec and RNNLM, their parameters are updated partially for one iteration. For example, the Word2Vec only updates the rows (or columns) of the weight matrix corresponding to words that appear in the current processing sentence.

      Currently, when the worker calls Update() function for a Param object. All its its gradients are sent to the server and all its values are updated. When applied to Word2Vec model, this would cause a big overhead because most parameters in Param object are has gradients (or with gradient 0), hence should not be updated.

      To support sparse update, we can create a SparseParam which subclasses the Param class. It provides APIs for users to set the updating area (e.g., the range using offset and length, or columns or rows) after computing the gradients in Layer's ComptueGradient function. The Worker's Update function is not changed, but the SparseParam will override the message generating functions to send gradients of the updating area.

      Due to parameter sharing, the Stub may also need updates to consider this case when doing local aggregation. The Updater should also work only on the update area.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            wangwei.cs wangwei

            Dates

              Created:
              Updated:

              Slack

                Issue deployment