[SINGA-407] Singa example used up all memory and hangs

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Application
• Assignee: Unassigned
• Reporter: Yin Xu
• Created: 19/Nov/18 06:54
• Updated: 23/Nov/18 14:59

Description

I installed singa on my machine and run the exapmles

https://github.com/apache/incubator-singa/tree/master/examples/cifar10

I simply run python train.py vgg cifar-10-batches-py.

It runs fine initially, but it keep using the memory and finally used up all the memory and swap, then the machine hangs.

My machine is Ubuntu 18.04, with kernel 4.15.0-39-generic

GPU card is: GeForce GTX 1060

The singa version is 1.2.0 py36_cuda9.0_cudnn7.1.2

Attach I show the GPU and resource usage when it hangs