[SINGA-46] Fix a bug in updater.cc to scale the gradients

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Unassigned
• Reporter: wangwei
• Created: 08/Aug/15 13:34
• Updated: 13/Aug/15 05:13
• Resolved: 13/Aug/15 05:13

Description

The Updater's Update function has an argument grad_scale (default value is 1.0), which is used to scale the gradients of parameters. For instance, when n workers compute over one mini-batch (each worker is assigned 1/n records), then their gradients should be averaged. We can do the average by passing grad_scale=1/n.

Some updaters in updater.cc, e.g., AdaGradUpdater, forget to scale the gradients by grad_scale. The bug can be fixed by

if (grad_scale != 1)
    grad *= grad_scale;