Details
-
Umbrella
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
4.0.0
-
None
-
None
Description
currently we only have single testing image (dev/infra/Dockerfile), for jobs pyspark, sparkr, lint and docs, it has two major issues:
- disk space limitation: we are adding more and more packages in it, the disk space left for testing is very limited, and cause No space left on device from time to time;
- environment conflicts: for example, even though we already install some packages for docs in the docker file, we still need to install some additional python packages in build_and_test, due to the conflicts between docs and pyspark. It is hard to maintain because the related packages are installed in different places.
so we want to split existing base image to multiple ones, so that:
- completely cache all the dependencies for each job;
- centralize related installations for each job;
- free up disk space on the base image;
- introduce new dev tools based on new images;