Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
Description
In madlib_keras dev-check, we create the input data to fit using minibatch_preprocessor_dl(). This function internally calls array_scalar_mult(). If we call either of these functions followed by madlib_keras_fit(), then the following error pops up:
NOTICE: Releasing segworker groups to finish aborting the transaction. ERROR: could not connect to segment: initialization of segworker group failed (cdbgang.c:237)
Digging further into Postgres logs suggests that there was a segmentation fault, and it seems like it's happening the moment import keras is called in madlib_keras_fit().
This issue was first noticed while working on MADLIB-1304 (which was closed with this commit, as the comment in this line suggests. This happened on Greenplum then, and Postgres was not supporting deep learning yet. This was again noticed while working on MADLIB-1311, which added Postgres support. At this point, the failure happened on Postgres and there were no failures on Greenplum.
While working on MADLIB-1311, we tried a couple of things and observed an odd behavior. We created a dummy function:
create function dummy()
returns void as
$$
import keras
$$
language plpythonu;
If we ran select dummy() before running minibatch_preprocessor_dl() or array_scalar_mult(), then the whole dev-check passes. But running the same function right after calling either of those functions causes a failure.
So, looks like any UDF that calls import keras must be run before calling minibatch_preprocessor_dl() or array_scalar_mult().