[MXNET-99] P3 (Nvidia V100) not supported by CUDA8

• Type: Bug
• Status: Done
• Priority: Major
• Resolution: Unresolved
• Component/s: CI Build
• Assignee: Marco de Abreu
• Reporter: Marco de Abreu
• Created: 14/Mar/18 15:45
• Updated: 21/Mar/18 16:05

Description

During my tests to upgrade our CI for [Quantization support|https://github.com/apache/incubator-mxnet/pull/9552,] I have noticed that NVCC supplied with CUDA 8 does not support the GPU architecture 7.0, which represents Volta GPUs used in AWS P3 instances. Our Makefile automatically removes this architecture in https://github.com/apache/incubator-mxnet/blob/master/Makefile#L280, which is fine since it would crash otherwise.

 

This causes pre-compilation to generate incompatible PTX binaries and thus causing issues during runtime - the JIT compiler has to kick in and libraries like CuBLAS report CUBLAS_STATUS_ARCH_MISMATCH. 

 

Issue Links

• links to: GitHub Pull Request #10108