[MADLIB-1308] Change GPU related hardcoded things in madlib_keras.py_in

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v1.16
• Component/s: Deep Learning
• Assignee: Unassigned
• Reporter: Nandish Jayaram
• Created: 20/Mar/19 17:33
• Updated: 06/Jun/19 00:36
• Resolved: 06/Jun/19 00:36

Description

Based on the code in PR https://github.com/apache/madlib/pull/355:

1. Currently in madlib_keras.py_in , we hardcod the following things
   1. gpus_per_host = 4
   2. device_name = '/cpu:0' or '/gpu:0' ( can the device ever be not named gpu0 or cpu0 ? )
2. Look into and document the usage of CUDA_VISIBLE_DEVICES when gpu_only is set to TRUE. Currently we set it to str(current_seg_id % gpus_per_host). How does this logic work and will it always work? How would this logic change for Postgres, since it has no segment_id.
3. What happens if
   no of gpus < no of segments
   no of gpus > no of segments