Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3611 Support Docker Containers In LinuxContainerExecutor
  3. YARN-8485

Priviledged container app launch is failing intermittently

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0, 3.1.1
    • 3.2.0, 3.1.1
    • yarn-native-services
    • Debian

    Description

      Privileged application fails intermittently 

      yarn  jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-*.jar  -shell_command "sleep 30" -num_containers 1 -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=xxx -shell_env YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER=true -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-*.jar

      Here, container launch fails with 'Privileged containers are disabled' even though Docker privilege container is enabled in the cluster

      nm log
      2018-06-28 21:21:15,647 INFO  runtime.DockerLinuxContainerRuntime (DockerLinuxContainerRuntime.java:allowPrivilegedContainerExecution(664)) - All checks pass. Launching privileged container for : container_e01_1530220647587_0001_01_000002
      2018-06-28 21:21:15,665 WARN  nodemanager.LinuxContainerExecutor (LinuxContainerExecutor.java:handleExitCode(593)) - Exit code from container container_e01_1530220647587_0001_01_000002 is : 29
      2018-06-28 21:21:15,666 WARN  nodemanager.LinuxContainerExecutor (LinuxContainerExecutor.java:handleExitCode(599)) - Exception from container-launch with container ID: container_e01_1530220647587_0001_01_000002 and exit code: 29
      org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: Launch container failed
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:958)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:141)
              at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleLaunchForLaunchType(LinuxContainerExecutor.java:564)
              at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:479)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:494)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:306)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:103)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      2018-06-28 21:21:15,668 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Exception from container-launch.
      2018-06-28 21:21:15,668 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Container id: container_e01_1530220647587_0001_01_000002
      2018-06-28 21:21:15,668 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Exit code: 29
      2018-06-28 21:21:15,668 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Exception message: Launch container failed
      2018-06-28 21:21:15,668 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Shell error output: check privileges failed for user: hrt_qa, error code: 0
      2018-06-28 21:21:15,668 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Privileged containers are disabled for user: hrt_qa
      2018-06-28 21:21:15,668 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Error constructing docker command, docker error code=11, error message='Privileged containers are disabled'
      2018-06-28 21:21:15,668 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) -
      2018-06-28 21:21:15,669 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Shell output: main : command provided 4
      2018-06-28 21:21:15,669 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - main : run as user is hrt_qa
      2018-06-28 21:21:15,669 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - main : requested yarn user is hrt_qa
      2018-06-28 21:21:15,669 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Creating script paths...
      2018-06-28 21:21:15,669 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Creating local dirs...
      2018-06-28 21:21:15,669 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Creating script paths...
      2018-06-28 21:21:15,669 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541)) - Creating local dirs...
      2018-06-28 21:21:15,693 WARN  launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(598)) - Container launch failed : Container exited with a non-zero exit code 29.
      2018-06-28 21:21:15,693 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(623)) - Failed to get tail of the container's prelaunch error log file
      java.io.FileNotFoundException: File /grid/0/hadoop/yarn/log/application_1530220647587_0001/container_e01_1530220647587_0001_01_000002/prelaunch.err does not exist
              at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
              at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
              at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitWithFailure(ContainerLaunch.java:609)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitCode(ContainerLaunch.java:575)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:340)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:103)  
               at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      2018-06-28 21:21:15,704 INFO  container.ContainerImpl (ContainerImpl.java:handle(2093)) - Container container_e01_1530220647587_0001_01_000002 transitioned from RUNNING to EXITED_WITH_FAILURE

      Attachments

        1. YARN-8485.001.patch
          1 kB
          Eric Yang
        2. YARN-8485.002.patch
          1 kB
          Eric Yang

        Issue Links

          Activity

            People

              eyang Eric Yang
              yeshavora Yesha Vora
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: