Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Currently for postgres, we create a keras session in the fit function and keep it open for all the iterations and then close it at the end.
We did a few experiments and found out that if we create 1 keras session per iteration , it performs slightly better.
We also found a bug in postgres deep learning madlib UDFs
If gpus are available on the host but the user passed in 0 for the gpus_per_host param, we still end up using gpu memory. This is because in the madlib_keras.fit UDF we call model.get_weights() to initialize the model if transfer learning and warm start are not applicable, and this function ends up using gpu if there are any available.