Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Cannot Reproduce
-
None
-
None
Description
When I train a model whose size is about 400MB, the execution time for serialize_nd_weights can be very slow. Specifically, the first-time execution of serialize_nd_weights lasts about 2 seconds, which is reasonable. However, afterward, its execution time becomes about 190 seconds.
I think the memory shortage causes the long execution time for serialize_nd_weights, since the time can be reduced with a more memory-efficient implementation for serializing (from 190 seconds to 70 seconds).
However, the instance I run Madlib has 290G memory available. I think that Madlib might set the wrong memory limit for its task. Is there any way to configure the memory limit for Madlib tasks?