[SYSTEMDS-2018] Fixing Weight Decay Regularization in ADAM

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Algorithms
• Assignee: Unassigned
• Reporter: Janardhan Pulivarthi
• Created: 16/Nov/17 08:19
• Updated: 20/Nov/17 19:03

Description

The common implementations of adaptive gradient algorithms, such
as Adam, limit the potential benefit of weight decay regularization, because the
weights do not decay multiplicatively (as would be expected for standard weight
decay) but by an additive constant factor.

This following paper found a way to fix regularization in Adam Optimization with one addition step(+ wx) to the gradient step :
https://arxiv.org/pdf/1711.05101.pdf