[MADLIB-1337] DL: Better warning and default for gpu memory fraction when no of gpus < no of segments

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: v1.16
• Component/s: Deep Learning
• Assignee: Unassigned
• Reporter: Nikhil
• Created: 02/May/19 18:01
• Updated: 14/Jun/19 16:22
• Resolved: 14/Jun/19 16:22

Description

We support the use case when no of gpus < no of segments however we noticed that sometimes this causes gpdb failures like

could not connect to segment: initialization of segworker group failed

1. We should give a meaningful warning to the user to make them aware that this feature may or may not work and also make a recommendation
2. We should also come up with a better heuristic for the memory fraction value. Currently we default to using 90% of the available memory and distribute it evenly among the segments.

Possible recommendations
1. Use as many gpus as segments (this may not be practical)
2. May be a smaller buffer size will help. Use minibatch preprocessor dl to pack less images. (we need to test this before we recommend it)