[MXNET-1374] Extensible for distributed training

• Type: New Feature
• Status: To Do
• Priority: Minor
• Resolution: Unresolved
• Component/s: Gluon
• Epic Link: Gluon Fit API
• Assignee: Unassigned
• Reporter: Lai Wei
• Created: 27/Mar/19 12:47
• Updated: 27/Mar/19 12:47

Description

1. Horovod: use a custom trainer
2. Parameter Server: batch_fn, trainer.step, should be the same as single node multi-GPU
3. consider on the convention to do mean(loss) and step(1) or step(batch_size), batch_size in Horovod is per device, in PS is per worker