[MADLIB-1310] Change the merge code to factor in the no of samples/images per buffer

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v1.16
• Component/s: Deep Learning
• Assignee: Unassigned
• Reporter: Nandish Jayaram
• Created: 20/Mar/19 17:38
• Updated: 02/May/19 19:02
• Resolved: 02/May/19 19:02

Description

Based on the code in PR https://github.com/apache/madlib/pull/355:

Right now while merging the weights, accuracy and loss, we assume that all the buffers are of the same size. This may not be true. Ideally we should should use the total no of images per segment to average. This can be done by adding the no of images to the state and then use this during merge to average.
This can be done by replacing the `buffer_count` variable in state with `image_count`.

Also investigate if there is a better way to pass SD other than the current logic i.e.

SD = kwargs['SD']