Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3611

Support Docker Containers In LinuxContainerExecutor

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: yarn
    • Labels:

      Description

      Support Docker Containers In LinuxContainerExecutor

      LinuxContainerExecutor provides useful functionality today with respect to localization, cgroups based resource management and isolation for CPU, network, disk etc. as well as security with a well-defined mechanism to execute privileged operations using the container-executor utility. Bringing docker support to LinuxContainerExecutor lets us use all of this functionality when running docker containers under YARN, while not requiring users and admins to configure and use a different ContainerExecutor.

      There are several aspects here that need to be worked through :

      • Mechanism(s) to let clients request docker-specific functionality - we could initially implement this via environment variables without impacting the client API.
      • Security - both docker daemon as well as application
      • Docker image localization
      • Running a docker container via container-executor as a specified user
      • “Isolate” the docker container in terms of CPU/network/disk/etc
      • Communicating with and/or signaling the running container (ensure correct pid handling)
      • Figure out workarounds for certain performance-sensitive scenarios like HDFS short-circuit reads
      • All of these need to be achieved without changing the current behavior of LinuxContainerExecutor

        Attachments

          Issue Links

          1.
          Add support for container runtimes in YARN Sub-task Resolved Sidharta Seethana
          2.
          Add docker container support to container-executor Sub-task Resolved Abin Shahab
          3.
          Add docker container runtime support to LinuxContainterExecutor Sub-task Resolved Sidharta Seethana
          4.
          Add localization support for docker images Sub-task Open Shane Kumpf
          5.
          container-executor should print output of docker logs if the docker container exits with non-0 exit status Sub-task Resolved Varun Vasudev
          6.
          Add support for different network setups when launching the docker container Sub-task Resolved Sidharta Seethana
          7.
          Add support for controlling capabilities for docker containers Sub-task Resolved Sidharta Seethana
          8.
          container-executor does not clean up docker operation command files. Sub-task Resolved Sidharta Seethana
          9.
          Allow whitelisted users to run privileged docker containers. Sub-task Resolved Sidharta Seethana
          10.
          Allow users to enter containers as UID:GID pair instead of by username Sub-task Resolved luhuichun
          11.
          Add additional logging to container launch implementations in container-executor Sub-task Resolved Sidharta Seethana
          12.
          Add cgroups support for docker containers Sub-task Resolved Sidharta Seethana
          13.
          Directories that are mounted in docker containers need to be more restrictive/container-specific Sub-task Resolved Sidharta Seethana
          14.
          Add support for configurable read-only mounts when launching Docker containers Sub-task Resolved Billie Rinaldi
          15.
          Container recovery is broken with delegating container runtime Sub-task Resolved Sidharta Seethana
          16.
          Fix signal handling for docker containers Sub-task Resolved Shane Kumpf
          17.
          Mount /sys/fs/cgroup into Docker containers as read only mount Sub-task Resolved luhuichun
          18.
          Add port mapping handling when docker container use bridge network Sub-task Open Unassigned
          19.
          Document Use of Docker with LinuxContainerExecutor Sub-task Resolved Daniel Templeton
          20.
          Mount usercache and NM filecache directories into Docker container Sub-task Resolved Sidharta Seethana
          21.
          Log Docker run command when container fails Sub-task Resolved Varun Vasudev
          22.
          Decouple host user and Docker container user Sub-task Resolved Zhankun Tang
          23.
          Improve handling of the Docker container life cycle Sub-task Resolved Shane Kumpf
          24.
          Remove bind-mount /etc/passwd for Docker containers Sub-task Resolved Zhankun Tang
          25.
          Allow for specifying the docker client configuration directory Sub-task Resolved Shane Kumpf
          26.
          Add support for docker inspect command Sub-task Resolved Shane Kumpf
          27.
          Add support for docker rm Sub-task Resolved Shane Kumpf
          28.
          Rename DockerStopCommandTest to TestDockerStopCommand Sub-task Resolved Shane Kumpf
          29.
          Allow user provided Docker volume mount list Sub-task Resolved Shane Kumpf
          30.
          Fix failing unit test in TestDockerContainerRuntime Sub-task Resolved Sidharta Seethana
          31.
          Remove package line length checkstyle rule Sub-task Resolved Shane Kumpf
          32.
          Support a white list of admin approved mounts Sub-task Resolved Shane Kumpf
          33.
          Add support for Docker pull Sub-task Resolved luhuichun
          34.
          Add support for Docker image clean up Sub-task Open Shane Kumpf
          35.
          Add support for Docker image clean up Sub-task Resolved Unassigned
          36.
          Support the Docker Live Restore feature Sub-task Resolved Shane Kumpf
          37.
          Improve signaling of short lived containers Sub-task Resolved Shane Kumpf
          38.
          Improve test coverage and add utility classes for common Docker operations Sub-task Resolved Shane Kumpf
          39.
          Improve Diagonstic by moving Error stack trace from NM to slider AM Sub-task Open Unassigned
          40.
          add mounting of HDFS Short-Circuit path for docker containers Sub-task Resolved Jaeboo Jeong
          41.
          check docker container's exit code when writing to cgroup task files Sub-task Patch Available Jaeboo Jeong
          42.
          Add default volume mount list Sub-task Resolved Eric Badger
          43.
          Add support to turn off launching privileged containers in the container-executor Sub-task Resolved Varun Vasudev
          44.
          container-executor fails for docker when command length > 4096 B Sub-task Resolved Eric Badger
          45.
          Make CGROUPS_ROOT_DIRECTORY configurable in DockerLinuxContainerRuntime Sub-task Resolved Eric Badger
          46.
          Add ability to specify volumes to mount for DockerContainerRuntime Sub-task Resolved Eric Yang
          47.
          Container-executor doesn't remove Docker containers that error out early Sub-task Resolved Eric Badger
          48.
          Add support for a volume blacklist for docker containers Sub-task Open Unassigned
          49.
          Make Docker target directory for cgroups configurable by yarn-site.xml Sub-task Resolved Shane Kumpf
          50.
          Consider /sys/fs/cgroup as the default CGroup mount path Sub-task Resolved Shane Kumpf
          51.
          Make the cgroup mount into Docker containers configurable Sub-task Resolved Shane Kumpf
          52.
          Add support for docker to have no capabilities Sub-task Resolved Eric Badger
          53.
          Docker permitted volumes don't properly check for directories Sub-task Resolved Eric Badger
          54.
          NM fails to successfully kill tasks that run over their memory limit Sub-task Resolved Unassigned
          55.
          Fix issues with docker commands executed by container-executor Sub-task Resolved Shane Kumpf
          56.
          Trim configuration values in DockerLinuxContainerRuntime Sub-task Resolved Tianyin Xu
          57.
          Docker container privileged mode and --user flag contradict each other Sub-task Resolved Eric Yang
          58.
          Enable user re-mapping for Docker containers by default Sub-task Resolved Eric Yang
          59.
          Security check for trusted docker image Sub-task Resolved Eric Yang
          60.
          NM gets backed up deleting docker containers Sub-task Open Eric Badger
          61.
          Support ENTRY_POINT for docker container Sub-task Resolved Eric Yang
          62.
          Docker Stop grace period should be configurable Sub-task Resolved Eric Badger
          63.
          Avoid using docker volume --format option to run against older docker releases Sub-task Resolved Wangda Tan
          64.
          Add support for setting the PID namespace mode Sub-task Resolved Billie Rinaldi
          65.
          Enable user re-mapping for Docker containers in yarn-default.xml Sub-task Resolved Eric Yang
          66.
          Docker host network can not obtain IP address for RegistryDNS Sub-task Resolved Eric Yang
          67.
          TestDockerContainerRuntime test failures due to UID lookup of a non-existent user Sub-task Resolved Shane Kumpf
          68.
          Remove automatic mounting of the cgroups root directory into Docker containers Sub-task Resolved Shane Kumpf
          69.
          Make the YARN mounts added to Docker containers more restrictive Sub-task Resolved Shane Kumpf
          70.
          Force removal of docker containers that do not get removed on first try Sub-task Open Unassigned
          71.
          Allow administrators to set a single ContainerRuntime for all containers Sub-task Open Unassigned
          72.
          Privileged, trusted containers need all of their bind-mounted directories to be read-only Sub-task Open Unassigned
          73.
          Fix exit code handling for short lived Docker containers Sub-task Resolved Shane Kumpf
          74.
          Trusted image log message repeated multiple times Sub-task Resolved Shane Kumpf
          75.
          Remove call to docker logs on failure in container-executor Sub-task Resolved Shane Kumpf
          76.
          Fix failing test TestDockerContainerRuntime#testLaunchContainerWithDockerTokens Sub-task Resolved Shane Kumpf
          77.
          Expose container's hostname to applications running within the docker container Sub-task Patch Available Suma Shivaprasad
          78.
          Add no-new-privileges flag to docker run Sub-task Resolved Eric Badger
          79.
          Support ContainerRelaunch for Docker containers Sub-task Resolved Shane Kumpf
          80.
          Docker container name(--name) needs to be DNS friendly for DNS resolution to work in user defined networks. Sub-task Resolved Suma Shivaprasad
          81.
          Add support for network-alias in docker run for user defined networks Sub-task Open Suma Shivaprasad
          82.
          YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS should not use commas as separators Sub-task Resolved Jim Brennan
          83.
          Setting hostname of docker container breaks for --net=host in docker 1.13 Sub-task Resolved Jim Brennan
          84.
          Docker launch fails when user private filecache directory is missing Sub-task Resolved Jason Lowe
          85.
          Allow regular expression matching in container-executor.cfg for devices and named docker volumes mount Sub-task Resolved Zian Chen
          86.
          Docker ".cmd" files should not be put in hadoop.tmp.dir Sub-task Resolved Eric Badger
          87.
          Add support for Docker env-file switch Sub-task Resolved Unassigned
          88.
          Sending a kill does not immediately kill docker containers Sub-task Resolved Eric Badger
          89.
          Docker container launch use popen have risk of shell expansion Sub-task Resolved Eric Yang
          90.
          Docker does not support hostnames greater than 64 characters Sub-task Resolved Shane Kumpf
          91.
          MRAppMaster fails when using UID:GID pair within docker container Sub-task Open Unassigned
          92.
          Revisit liveliness checks for Docker containers Sub-task Patch Available Shane Kumpf
          93.
          Docker container launch fails due to .cmd file creation failure Sub-task Resolved Jason Lowe
          94.
          DockerClient still touches hadoop.tmp.dir Sub-task Open Unassigned
          95.
          Remove unused environment variables from the Docker runtime Sub-task Resolved Eric Badger
          96.
          Update documentation and yarn-default related to the Docker runtime Sub-task Open Unassigned
          97.
          get_docker_command refactoring Sub-task Resolved Eric Badger
          98.
          Docker client configuration can still be set incorrectly Sub-task Patch Available Shane Kumpf
          99.
          Privileged docker containers' jobSubmitDir does not get successfully cleaned up Sub-task Resolved Unassigned
          100.
          YARN should have ability to run images only from a whitelist docker registries Sub-task Open Unassigned
          101.
          Using docker image from a non-privileged registry, the launch_command is not honored Sub-task Patch Available Eric Yang

            Activity

              People

              • Assignee:
                sidharta-s Sidharta Seethana
                Reporter:
                sidharta-s Sidharta Seethana
              • Votes:
                4 Vote for this issue
                Watchers:
                84 Start watching this issue

                Dates

                • Created:
                  Updated: