Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7913

test-with-docker can sometimes fail with a ccache fatal error when using CentOS 6 within the containers

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 3.1.0
    • Fix Version/s: Impala 3.2.0
    • Component/s: Infrastructure
    • Labels:
      None
    • Environment:
      test-with-docker
    • Epic Color:
      ghx-label-8

      Description

      When using docker/test-with-docker.py, Impala builds only the core Impala binaries in the "build" phase/container. Test and benchmark executables are built in later phases in the various test containers. These latter builds are executed in parallel in the context of concurrently running containers. These containers separate the running processes, but they share a common ccache directory for efficiency's sake; this directory is managed by the ccache processes running inside the containers.

      Recently I have seen multiple ccache failures in this phase, when the containers are hosting centos6 (test-with-docker.py launched with --base-image=centos:6). The failures usually look like this:

      2018-11-27 04:59:52.826775 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/RuntimeProfile_constants.cpp.o
      2018-11-27 04:59:52.933599 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/RuntimeProfile_types.cpp.o
      2018-11-27 04:59:53.041421 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/StatestoreService.cpp.o
      2018-11-27 04:59:53.169103 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/StatestoreService_constants.cpp.o
      2018-11-27 04:59:53.544171 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/StatestoreService_types.cpp.o
      2018-11-27 04:59:53.576886 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/StatestoreSubscriber.cpp.o
      2018-11-27 04:59:53.946981 ccache: FATAL: Failed to open /home/impdev/.ccache/tmp/tmp.cpp_stderr.i-20181127-033918.7545: No such file or directory
      2018-11-27 04:59:53.947712 make[3]: *** [be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/RuntimeProfile_constants.cpp.o] Error 1
      

      The issue seems to be caused by the cache being shared across containers, and running an older ccache version under CentOS 6:  https://ccache.samba.narkive.com/o4BSOjxG/shared-ccache-directory-between-docker-containers claims that ccache v3.1 (the one CentOS 6 pulls from EPEL by default) has a problem generating unique filenames for its temporary files.

        Attachments

          Activity

            People

            • Assignee:
              laszlog Laszlo Gaal
              Reporter:
              laszlog Laszlo Gaal
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: