[MADLIB-1418] DL: performance improvements to fit transition function

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v1.17
• Component/s: Deep Learning
• Assignee: Nikhil Kak
• Reporter: Nikhil Kak
• Created: 14/Mar/20 01:20
• Updated: 19/Mar/20 02:48
• Resolved: 19/Mar/20 02:48

Description

 

We can speed up our per row/buffer time by not serializing the model weights as part of state except for the last row. For all rows except the last row, all we need from the model state is just the agg_image_count, so we can just set the state to agg_image_count for all rows except the last row. For the last row, the behavior remains the same as before

{{}}

{{With places10 20 msts gpdb6 (with 1 epoch for all msts), per iteration time went down from around 1700-1800 secs to 1600-1700 secs. }}

This will be more pronounced when training with a bigger dataset like places365/imagenet.