[IMPALA-7913] test-with-docker can sometimes fail with a ccache fatal error when using CentOS 6 within the containers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.1.0
• Fix Version/s: Impala 3.2.0
• Component/s: Infrastructure
• Environment: test-with-docker
• Epic Color: ghx-label-8
• Assignee: Laszlo Gaal
• Reporter: Laszlo Gaal
• Created: 30/Nov/18 14:42
• Updated: 14/Dec/18 19:46
• Resolved: 14/Dec/18 19:46

Description

When using docker/test-with-docker.py, Impala builds only the core Impala binaries in the "build" phase/container. Test and benchmark executables are built in later phases in the various test containers. These latter builds are executed in parallel in the context of concurrently running containers. These containers separate the running processes, but they share a common ccache directory for efficiency's sake; this directory is managed by the ccache processes running inside the containers.

Recently I have seen multiple ccache failures in this phase, when the containers are hosting centos6 (test-with-docker.py launched with --base-image=centos:6). The failures usually look like this:

2018-11-27 04:59:52.826775 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/RuntimeProfile_constants.cpp.o
2018-11-27 04:59:52.933599 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/RuntimeProfile_types.cpp.o
2018-11-27 04:59:53.041421 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/StatestoreService.cpp.o
2018-11-27 04:59:53.169103 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/StatestoreService_constants.cpp.o
2018-11-27 04:59:53.544171 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/StatestoreService_types.cpp.o
2018-11-27 04:59:53.576886 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/StatestoreSubscriber.cpp.o
2018-11-27 04:59:53.946981 ccache: FATAL: Failed to open /home/impdev/.ccache/tmp/tmp.cpp_stderr.i-20181127-033918.7545: No such file or directory
2018-11-27 04:59:53.947712 make[3]: *** [be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/RuntimeProfile_constants.cpp.o] Error 1

The issue seems to be caused by the cache being shared across containers, and running an older ccache version under CentOS 6:  https://ccache.samba.narkive.com/o4BSOjxG/shared-ccache-directory-between-docker-containers claims that ccache v3.1 (the one CentOS 6 pulls from EPEL by default) has a problem generating unique filenames for its temporary files.