Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16311

Hadoop build failure - natively on ARM (armv7) - oom_listener_main.c issues

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 3.2.0
    • None
    • build
    • None

    • Linux ARM (armv7)

    Description

      After failing to build Hadoop 3.3.0 - latest git snapshot, described in:
      https://issues.apache.org/jira/browse/HADOOP-16309
      I went on and tried my luck with the 3.2.0 release, only to find out that I'm unable to build this version too. Environment is the same as described in:
      https://issues.apache.org/jira/browse/HADOOP-16309
      Just the source archive is different:
      https://archive.apache.org/dist/hadoop/common/hadoop-3.2.0/hadoop-3.2.0-src.tar.gz

      Hadoop build failure log snippet:

      [WARNING] make[2]: Leaving directory '/kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native'
      [WARNING] [ 44%] Built target container
      [WARNING] [ 47%] Linking CXX static library libgtest.a
      [WARNING] /usr/bin/cmake -P CMakeFiles/gtest.dir/cmake_clean_target.cmake
      [WARNING] /usr/bin/cmake -E cmake_link_script CMakeFiles/gtest.dir/link.txt --verbose=1
      [WARNING] /usr/bin/ar qc libgtest.a CMakeFiles/gtest.dir/kit/hadoop-3.2.0-src/hadoop-common-project/hadoop-common/src/main/native/gtest/gtest-all.cc.o
      [WARNING] /usr/bin/ranlib libgtest.a
      [WARNING] make[2]: Leaving directory '/kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native'
      [WARNING] [ 47%] Built target gtest
      [WARNING] make[1]: Leaving directory '/kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native'
      [WARNING] /usr/bin/ld: CMakeFiles/oom-listener.dir/main/native/oom-listener/impl/oom_listener_main.c.o: in function `main':
      [WARNING] /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/impl/oom_listener_main.c:89: undefined reference to `cleanup'
      [WARNING] collect2: error: ld returned 1 exit status
      [WARNING] make[2]: *** [CMakeFiles/oom-listener.dir/build.make:99: target/usr/local/bin/oom-listener] Error 1
      [WARNING] make[1]: *** [CMakeFiles/Makefile2:222: CMakeFiles/oom-listener.dir/all] Error 2
      [WARNING] make[1]: *** Waiting for unfinished jobs....
      [WARNING] /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/gpu/gpu-module.c: In function ‘handle_gpu_request’:
      [WARNING] /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/gpu/gpu-module.c:196:9: warning: ‘strncpy’ specified bound 128 equals destination size [-Wstringop-truncation]
      [WARNING] 196 | strncpy(container_id, optarg, MAX_CONTAINER_ID_LEN);
      [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      [WARNING] /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/fpga/fpga-module.c: In function ‘handle_fpga_request’:
      [WARNING] /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/fpga/fpga-module.c:196:9: warning: ‘strncpy’ specified bound 128 equals destination size [-Wstringop-truncation]
      [WARNING] 196 | strncpy(container_id, optarg, MAX_CONTAINER_ID_LEN);
      [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      [WARNING] /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c: In function ‘normalize_mount’:
      [WARNING] /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1041:9: warning: ‘strncpy’ output truncated before terminating nul copying as many bytes from a string as its length [-Wstringop-truncation]
      [WARNING] 1041 | strncpy(ret_ptr, real_mount, len);
      [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      [WARNING] /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1034:20: note: length computed here
      [WARNING] 1034 | size_t len = strlen(real_mount);
      [WARNING] | ^~~~~~~~~~~~~~~~~~
      [WARNING] /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c: In function ‘check_trusted_image’:
      [WARNING] /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:128:9: warning: ‘strncpy’ output truncated before terminating nul copying as many bytes from a string as its length [-Wstringop-truncation]
      [WARNING] 128 | strncpy(registry_ptr, privileged_registry[i], len);
      [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      [WARNING] /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:125:17: note: length computed here
      [WARNING] 125 | int len = strlen(privileged_registry[i]);
      [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      [WARNING] /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c: In function ‘mount_cgroup’:
      [WARNING] /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:2445:19: warning: ‘stpncpy’ output truncated before terminating nul copying as many bytes from a string as its length [-Wstringop-truncation]
      [WARNING] 2445 | char *buf = stpncpy(hier_path, mount_path, strlen(mount_path));
      [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      [WARNING] make: *** [Makefile:84: all] Error 2
      [INFO] ------------------------------------------------------------------------
      [INFO] Reactor Summary for Apache Hadoop Main 3.2.0:
      [INFO]
      [INFO] Apache Hadoop Main ................................. SUCCESS [ 9.998 s]
      ...
      [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [03:56 min]
      [INFO] Apache Hadoop YARN NodeManager ..................... FAILURE [04:08 min]
      [INFO] Apache Hadoop YARN Web Proxy ....................... SKIPPED
      ...
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 03:03 h
      [INFO] Finished at: 2019-05-13T05:06:13+03:00
      [INFO] ------------------------------------------------------------------------
      [ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on project hadoop-yarn-server-nodemanager: make failed with error code 2 -> [Help 1]

      Did some investigation, went for a manual compilation of the affected source tree:
      /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native# make -j 4 V=1
      Scanning dependencies of target oom-listener
      [ 5%] Built target gtest
      [ 38%] Built target container
      [ 41%] Building C object CMakeFiles/oom-listener.dir/main/native/oom-listener/impl/oom_listener_main.c.o
      [ 44%] Building CXX object CMakeFiles/test-oom-listener.dir/main/native/oom-listener/test/oom_listener_test_main.cc.o
      [ 50%] Built target container-executor
      [ 52%] Building CXX object CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o
      [ 58%] Built target test-container-executor
      [ 61%] Linking C executable target/usr/local/bin/oom-listener
      /usr/bin/ld: CMakeFiles/oom-listener.dir/main/native/oom-listener/impl/oom_listener_main.c.o: in function `main':
      /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/impl/oom_listener_main.c:89: undefined reference to `cleanup'
      collect2: error: ld returned 1 exit status
      make[2]: *** [CMakeFiles/oom-listener.dir/build.make:99: target/usr/local/bin/oom-listener] Error 1
      make[1]: *** [CMakeFiles/Makefile2:222: CMakeFiles/oom-listener.dir/all] Error 2
      make[1]: *** Waiting for unfinished jobs....
      /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/test/oom_listener_test_main.cc: In member function ‘virtual void OOMListenerTest_test_oom_Test::TestBody()’:
      /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/test/oom_listener_test_main.cc:256:7: error: ‘__WAIT_STATUS’ was not declared in this scope; did you mean ‘ADJ_STATUS’?
      256 | __WAIT_STATUS mem_hog_status = {};

      ^~~~~~~~~~~~~
      ADJ_STATUS
      /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/test/oom_listener_test_main.cc:257:30: error: ‘mem_hog_status’ was not declared in this scope
      257       		__pid_t exited0 = wait(mem_hog_status);
      ^~~~~~~~~~~~~~
      /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/test/oom_listener_test_main.cc:275:20: error: expected ‘;’ before ‘oom_listener_status’
      275       		__WAIT_STATUS oom_listener_status = {};
      ^~~~~~~~~~~~~~~~~~~~
      ;
      /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/test/oom_listener_test_main.cc:276:30: error: ‘oom_listener_status’ was not declared in this scope; did you mean ‘oom_listener’?
      276       		__pid_t exited1 = wait(oom_listener_status);
      ^~~~~~~~~~~~~~~~~~~
      oom_listener
      make[2]: *** [CMakeFiles/test-oom-listener.dir/build.make:76: CMakeFiles/test-oom-listener.dir/main/native/oom-listener/test/oom_listener_test_main.cc.o] Error 1
      make[1]: *** [CMakeFiles/Makefile2:73: CMakeFiles/test-oom-listener.dir/all] Error 2
      /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc: In member function ‘char* ContainerExecutor::TestDockerUtil::flatten(args*)’:
      /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:107:20 warning: ‘char* strncpy(char*, const char*, size_t)’ output truncated before terminating nul copying 1 byte from a string of the same length [-Wstringop-truncation]
      107       		strncpy(buffer + current_len, " ", 1);
      ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      make[2]: *** wait: No child processes. Stop.
      make[2]: *** Waiting for unfinished jobs....

      Then at line 89 in
      hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/impl/oom_listener_main.c

      • commented:
        /*  cleanup(&descriptors); */

      Tried a manual compilation again and got into more errors:
      /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native# make -j 4 V=1
      Scanning dependencies of target oom-listener
      [ 5%] Built target gtest
      [ 8%] Building C object CMakeFiles/oom-listener.dir/main/native/oom-listener/impl/oom_listener_main.c.o
      [ 41%] Built target container
      [ 44%] Building CXX object CMakeFiles/test-oom-listener.dir/main/native/oom-listener/test/oom_listener_test_main.cc.o
      [ 50%] Built target container-executor
      [ 52%] Building CXX object CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o
      [ 58%] Built target test-container-executor
      [ 61%] Linking C executable target/usr/local/bin/oom-listener
      [ 63%] Built target oom-listener
      /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/test/oom_listener_test_main.cc: In member function ‘virtual void OOMListenerTest_test_oom_Test::TestBody()’:
      /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/test/oom_listener_test_main.cc:256:7: error: ‘__WAIT_STATUS’ was not declared in this scope; did you mean ‘ADJ_STATUS’?
      256 | __WAIT_STATUS mem_hog_status = {};

      ^~~~~~~~~~~~~
      ADJ_STATUS
      /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/test/oom_listener_test_main.cc:257:30: error: ‘mem_hog_status’ was not declared in this scope
      257       		__pid_t exited0 = wait(mem_hog_status);
      ^~~~~~~~~~~~~~
      /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/test/oom_listener_test_main.cc:275:20: error: expected ‘;’ before ‘oom_listener_status’
      275       		__WAIT_STATUS oom_listener_status = {};
      ^~~~~~~~~~~~~~~~~~~~
      ;
      /kit/hadoop-3.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/test/oom_listener_test_main.cc:276:30: error: ‘oom_listener_status’ was not declared in this scope; did you mean ‘oom_listener’?
      276       		__pid_t exited1 = wait(oom_listener_status);
      ^~~~~~~~~~~~~~~~~~~
      oom_listener
      make[2]: *** [CMakeFiles/test-oom-listener.dir/build.make:76: CMakeFiles/test-oom-listener.dir/main/native/oom-listener/test/oom_listener_test_main.cc.o] Error 1
      make[1]: *** [CMakeFiles/Makefile2:73: CMakeFiles/test-oom-listener.dir/all] Error 2
      make[1]: *** Waiting for unfinished jobs....

      Attachments

        Activity

          People

            Unassigned Unassigned
            abga AiBe Gee
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: