To run the tests, set the docker-service-url to the host and port where
- * docker service is running (If docker-service-url is not specified then the
- * local daemon will be used).
- *
- * mvn test -Ddocker-service-url=tcp://0.0.0.0:4243 -Dtest=TestDockerContainerExecutor
- *
- */
-public class TestDockerContainerExecutor {
- private static final Log LOG = LogFactory
- .getLog(TestDockerContainerExecutor.class);
- private static File workSpace = null;
- private DockerContainerExecutor exec = null;
- private LocalDirsHandlerService dirsHandler;
- private Path workDir;
- private FileContext lfs;
- private String yarnImage;
-
- private String appSubmitter;
- private String dockerUrl;
- private String testImage = "centos:latest";
- private String dockerExec;
- private ContainerId getNextContainerId() {
- ContainerId cId = mock(ContainerId.class, RETURNS_DEEP_STUBS);
- String id = "CONTAINER_" + System.currentTimeMillis();
- when(cId.toString()).thenReturn(id);
- return cId;
- }
-
- @Before
- //Initialize a new DockerContainerExecutor that will be used to launch mocked
- //containers.
- public void setup() {
- try {
- lfs = FileContext.getLocalFSFileContext();
- workDir = new Path("/tmp/temp-" + System.currentTimeMillis());
- workSpace = new File(workDir.toUri().getPath());
- lfs.mkdir(workDir, FsPermission.getDirDefault(), true);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- Configuration conf = new Configuration();
- yarnImage = "yarnImage";
- long time = System.currentTimeMillis();
- conf.set(YarnConfiguration.NM_LOCAL_DIRS, "/tmp/nm-local-dir" + time);
- conf.set(YarnConfiguration.NM_LOG_DIRS, "/tmp/userlogs" + time);
-
- dockerUrl = System.getProperty("docker-service-url");
- LOG.info("dockerUrl: " + dockerUrl);
- if (!Strings.isNullOrEmpty(dockerUrl)) {
- dockerUrl = " -H " + dockerUrl;
- } else if(isDockerDaemonRunningLocally()) {
- dockerUrl = "";
- } else {
- return;
- }
- dockerExec = "docker " + dockerUrl;
- conf.set(
- YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, yarnImage);
- conf.set(
- YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME, dockerExec);
- exec = new DockerContainerExecutor();
- dirsHandler = new LocalDirsHandlerService();
- dirsHandler.init(conf);
- exec.setConf(conf);
- appSubmitter = System.getProperty("application.submitter");
- if (appSubmitter == null || appSubmitter.isEmpty()) {
- appSubmitter = "nobody";
- }
- shellExec(dockerExec + " pull " + testImage);
-
- }
-
- private Shell.ShellCommandExecutor shellExec(String command) {
- try {
- Shell.ShellCommandExecutor shExec = new Shell.ShellCommandExecutor(
- command.split("\\s+"),
- new File(workDir.toUri().getPath()),
- System.getenv());
- shExec.execute();
- return shExec;
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
-
- private boolean shouldRun() {
- return exec != null;
- }
-
- private boolean isDockerDaemonRunningLocally() {
- boolean dockerDaemonRunningLocally = true;
- try {
- shellExec("docker info");
- } catch (Exception e) {
- LOG.info("docker daemon is not running on local machine.");
- dockerDaemonRunningLocally = false;
- }
- return dockerDaemonRunningLocally;
- }
-
- /**
- * Test that a docker container can be launched to run a command
- * @param cId a fake ContainerID
- * @param launchCtxEnv
- * @param cmd the command to launch inside the docker container
- * @return the exit code of the process used to launch the docker container
- * @throws IOException
- */
- private int runAndBlock(ContainerId cId, Map launchCtxEnv,
- String... cmd) throws IOException {
- String appId = "APP_" + System.currentTimeMillis();
- Container container = mock(Container.class);
- ContainerLaunchContext context = mock(ContainerLaunchContext.class);
-
- when(container.getContainerId()).thenReturn(cId);
- when(container.getLaunchContext()).thenReturn(context);
- when(cId.getApplicationAttemptId().getApplicationId().toString())
- .thenReturn(appId);
- when(context.getEnvironment()).thenReturn(launchCtxEnv);
-
- String script = writeScriptFile(launchCtxEnv, cmd);
-
- Path scriptPath = new Path(script);
- Path tokensPath = new Path("/dev/null");
- Path workDir = new Path(workSpace.getAbsolutePath());
- Path pidFile = new Path(workDir, "pid.txt");
-
- exec.activateContainer(cId, pidFile);
- return exec.launchContainer(new ContainerStartContext.Builder()
- .setContainer(container)
- .setNmPrivateContainerScriptPath(scriptPath)
- .setNmPrivateTokensPath(tokensPath)
- .setUser(appSubmitter)
- .setAppId(appId)
- .setContainerWorkDir(workDir)
- .setLocalDirs(dirsHandler.getLocalDirs())
- .setLogDirs(dirsHandler.getLogDirs())
- .build());
- }
-
- // Write the script used to launch the docker container in a temp file
- private String writeScriptFile(Map launchCtxEnv,
- String... cmd) throws IOException {
- File f = File.createTempFile("TestDockerContainerExecutor", ".sh");
- f.deleteOnExit();
- PrintWriter p = new PrintWriter(new FileOutputStream(f));
- for(Map.Entry entry: launchCtxEnv.entrySet()) {
- p.println("export " + entry.getKey() + "=\"" + entry.getValue() + "\"");
- }
- for (String part : cmd) {
- p.print(part.replace("\\", "\\\\").replace("'", "\\'"));
- p.print(" ");
- }
- p.println();
- p.close();
- return f.getAbsolutePath();
- }
-
- @After
- public void tearDown() {
- try {
- lfs.delete(workDir, true);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
-
- /**
- * Test that a touch command can be launched successfully in a docker
- * container
- */
- @Test(timeout=1000000)
- public void testLaunchContainer() throws IOException {
- if (!shouldRun()) {
- LOG.warn("Docker not installed, aborting test.");
- return;
- }
-
- Map env = new HashMap();
- env.put(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME,
- testImage);
- String touchFileName = "touch-file-" + System.currentTimeMillis();
- File touchFile = new File(dirsHandler.getLocalDirs().get(0), touchFileName);
- ContainerId cId = getNextContainerId();
- int ret = runAndBlock(cId, env, "touch", touchFile.getAbsolutePath(), "&&",
- "cp", touchFile.getAbsolutePath(), "/");
-
- assertEquals(0, ret);
- }
-}
diff --git hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java
deleted file mode 100644
index 27ff438..0000000
--- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java
+++ /dev/null
@@ -1,305 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hadoop.yarn.server.nodemanager;
-
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assume.assumeTrue;
-import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.when;
-
-import java.io.File;
-import java.io.FileReader;
-import java.io.IOException;
-import java.io.LineNumberReader;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.CommonConfigurationKeys;
-import org.apache.hadoop.fs.FileContext;
-import org.apache.hadoop.fs.FileUtil;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.fs.permission.FsPermission;
-import org.apache.hadoop.util.Shell;
-import org.apache.hadoop.yarn.api.records.ContainerId;
-import org.apache.hadoop.yarn.api.records.ContainerLaunchContext;
-import org.apache.hadoop.yarn.conf.YarnConfiguration;
-import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
-import org.apache.hadoop.yarn.server.nodemanager.executor.ContainerStartContext;
-import org.junit.After;
-import org.junit.Before;
-import org.junit.Test;
-
-/**
- * Mock tests for docker container executor
- */
-public class TestDockerContainerExecutorWithMocks {
-
- private static final Log LOG = LogFactory
- .getLog(TestDockerContainerExecutorWithMocks.class);
- public static final String DOCKER_LAUNCH_COMMAND = "/bin/true";
- private DockerContainerExecutor dockerContainerExecutor = null;
- private LocalDirsHandlerService dirsHandler;
- private Path workDir;
- private FileContext lfs;
- private String yarnImage;
-
- @Before
- public void setup() {
- assumeTrue(Shell.LINUX);
- File f = new File("./src/test/resources/mock-container-executor");
- if(!FileUtil.canExecute(f)) {
- FileUtil.setExecutable(f, true);
- }
- String executorPath = f.getAbsolutePath();
- Configuration conf = new Configuration();
- yarnImage = "yarnImage";
- long time = System.currentTimeMillis();
- conf.set(YarnConfiguration.NM_LINUX_CONTAINER_EXECUTOR_PATH, executorPath);
- conf.set(YarnConfiguration.NM_LOCAL_DIRS, "/tmp/nm-local-dir" + time);
- conf.set(YarnConfiguration.NM_LOG_DIRS, "/tmp/userlogs" + time);
- conf.set(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME,
- yarnImage);
- conf.set(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME,
- DOCKER_LAUNCH_COMMAND);
- dockerContainerExecutor = new DockerContainerExecutor();
- dirsHandler = new LocalDirsHandlerService();
- dirsHandler.init(conf);
- dockerContainerExecutor.setConf(conf);
- lfs = null;
- try {
- lfs = FileContext.getLocalFSFileContext();
- workDir = new Path("/tmp/temp-"+ System.currentTimeMillis());
- lfs.mkdir(workDir, FsPermission.getDirDefault(), true);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
-
- @After
- public void tearDown() {
- try {
- if (lfs != null) {
- lfs.delete(workDir, true);
- }
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
-
- @Test(expected = IllegalStateException.class)
- //Test that DockerContainerExecutor doesn't successfully init on a secure
- //cluster
- public void testContainerInitSecure() throws IOException {
- dockerContainerExecutor.getConf().set(
- CommonConfigurationKeys.HADOOP_SECURITY_AUTHENTICATION, "kerberos");
- dockerContainerExecutor.init();
- }
-
- @Test(expected = IllegalArgumentException.class)
- //Test that when the image name is null, the container launch throws an
- //IllegalArgumentException
- public void testContainerLaunchNullImage() throws IOException {
- String appSubmitter = "nobody";
- String appId = "APP_ID";
- String containerId = "CONTAINER_ID";
- String testImage = "";
-
- Container container = mock(Container.class, RETURNS_DEEP_STUBS);
- ContainerId cId = mock(ContainerId.class, RETURNS_DEEP_STUBS);
- ContainerLaunchContext context = mock(ContainerLaunchContext.class);
-
- HashMap env = new HashMap();
- when(container.getContainerId()).thenReturn(cId);
- when(container.getLaunchContext()).thenReturn(context);
- when(cId.getApplicationAttemptId().getApplicationId().toString())
- .thenReturn(appId);
- when(cId.toString()).thenReturn(containerId);
-
- when(context.getEnvironment()).thenReturn(env);
- env.put(
- YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, testImage);
- dockerContainerExecutor.getConf().set(
- YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, testImage);
- Path scriptPath = new Path("file:///bin/echo");
- Path tokensPath = new Path("file:///dev/null");
-
- Path pidFile = new Path(workDir, "pid.txt");
-
- dockerContainerExecutor.activateContainer(cId, pidFile);
- dockerContainerExecutor.launchContainer(new ContainerStartContext.Builder()
- .setContainer(container)
- .setNmPrivateContainerScriptPath(scriptPath)
- .setNmPrivateTokensPath(tokensPath)
- .setUser(appSubmitter)
- .setAppId(appId)
- .setContainerWorkDir(workDir)
- .setLocalDirs(dirsHandler.getLocalDirs())
- .setLogDirs(dirsHandler.getLogDirs())
- .build());
- }
-
- @Test(expected = IllegalArgumentException.class)
- //Test that when the image name is invalid, the container launch throws an
- //IllegalArgumentException
- public void testContainerLaunchInvalidImage() throws IOException {
- String appSubmitter = "nobody";
- String appId = "APP_ID";
- String containerId = "CONTAINER_ID";
- String testImage = "testrepo.com/test-image rm -rf $HADOOP_HOME/*";
-
- Container container = mock(Container.class, RETURNS_DEEP_STUBS);
- ContainerId cId = mock(ContainerId.class, RETURNS_DEEP_STUBS);
- ContainerLaunchContext context = mock(ContainerLaunchContext.class);
- HashMap env = new HashMap();
-
- when(container.getContainerId()).thenReturn(cId);
- when(container.getLaunchContext()).thenReturn(context);
- when(cId.getApplicationAttemptId().getApplicationId().toString())
- .thenReturn(appId);
- when(cId.toString()).thenReturn(containerId);
-
- when(context.getEnvironment()).thenReturn(env);
- env.put(
- YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, testImage);
- dockerContainerExecutor.getConf().set(
- YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, testImage);
- Path scriptPath = new Path("file:///bin/echo");
- Path tokensPath = new Path("file:///dev/null");
-
- Path pidFile = new Path(workDir, "pid.txt");
-
- dockerContainerExecutor.activateContainer(cId, pidFile);
- dockerContainerExecutor.launchContainer(
- new ContainerStartContext.Builder()
- .setContainer(container)
- .setNmPrivateContainerScriptPath(scriptPath)
- .setNmPrivateTokensPath(tokensPath)
- .setUser(appSubmitter)
- .setAppId(appId)
- .setContainerWorkDir(workDir)
- .setLocalDirs(dirsHandler.getLocalDirs())
- .setLogDirs(dirsHandler.getLogDirs())
- .build());
- }
-
- @Test
- //Test that a container launch correctly wrote the session script with the
- //commands we expected
- public void testContainerLaunch() throws IOException {
- String appSubmitter = "nobody";
- String appId = "APP_ID";
- String containerId = "CONTAINER_ID";
- String testImage = "\"sequenceiq/hadoop-docker:2.4.1\"";
-
- Container container = mock(Container.class, RETURNS_DEEP_STUBS);
- ContainerId cId = mock(ContainerId.class, RETURNS_DEEP_STUBS);
- ContainerLaunchContext context = mock(ContainerLaunchContext.class);
- HashMap env = new HashMap();
-
- when(container.getContainerId()).thenReturn(cId);
- when(container.getLaunchContext()).thenReturn(context);
- when(cId.getApplicationAttemptId().getApplicationId().toString())
- .thenReturn(appId);
- when(cId.toString()).thenReturn(containerId);
-
- when(context.getEnvironment()).thenReturn(env);
- env.put(
- YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, testImage);
- Path scriptPath = new Path("file:///bin/echo");
- Path tokensPath = new Path("file:///dev/null");
-
- Path pidFile = new Path(workDir, "pid");
-
- dockerContainerExecutor.activateContainer(cId, pidFile);
- int ret = dockerContainerExecutor.launchContainer(
- new ContainerStartContext.Builder()
- .setContainer(container)
- .setNmPrivateContainerScriptPath(scriptPath)
- .setNmPrivateTokensPath(tokensPath)
- .setUser(appSubmitter)
- .setAppId(appId)
- .setContainerWorkDir(workDir)
- .setLocalDirs(dirsHandler.getLocalDirs())
- .setLogDirs(dirsHandler.getLogDirs())
- .build());
- assertEquals(0, ret);
- //get the script
- Path sessionScriptPath = new Path(workDir,
- Shell.appendScriptExtension(
- DockerContainerExecutor.DOCKER_CONTAINER_EXECUTOR_SESSION_SCRIPT));
- LineNumberReader lnr = new LineNumberReader(new FileReader(
- sessionScriptPath.toString()));
- boolean cmdFound = false;
- List localDirs = dirsToMount(dirsHandler.getLocalDirs());
- List logDirs = dirsToMount(dirsHandler.getLogDirs());
- List workDirMount = dirsToMount(Collections.singletonList(
- workDir.toUri().getPath()));
- List expectedCommands = new ArrayList(Arrays.asList(
- DOCKER_LAUNCH_COMMAND, "run", "--rm", "--net=host", "--name",
- containerId));
- expectedCommands.addAll(localDirs);
- expectedCommands.addAll(logDirs);
- expectedCommands.addAll(workDirMount);
- String shellScript = workDir + "/launch_container.sh";
-
- expectedCommands.addAll(Arrays.asList(testImage.replaceAll("['\"]", ""),
- "bash","\"" + shellScript + "\""));
-
- String expectedPidString =
- "echo `/bin/true inspect --format {{.State.Pid}} " + containerId+"` > "+
- pidFile.toString() + ".tmp";
- boolean pidSetterFound = false;
- while(lnr.ready()){
- String line = lnr.readLine();
- LOG.debug("line: " + line);
- if (line.startsWith(DOCKER_LAUNCH_COMMAND)){
- List command = new ArrayList();
- for( String s :line.split("\\s+")){
- command.add(s.trim());
- }
-
- assertEquals(expectedCommands, command);
- cmdFound = true;
- } else if (line.startsWith("echo")) {
- assertEquals(expectedPidString, line);
- pidSetterFound = true;
- }
-
- }
- assertTrue(cmdFound);
- assertTrue(pidSetterFound);
- }
-
- private List dirsToMount(List dirs) {
- List localDirs = new ArrayList();
- for(String dir: dirs){
- localDirs.add("-v");
- localDirs.add(dir + ":" + dir);
- }
- return localDirs;
- }
-}
diff --git hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainerExecutor.md.vm hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainerExecutor.md.vm
deleted file mode 100644
index 6a7db43..0000000
--- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainerExecutor.md.vm
+++ /dev/null
@@ -1,154 +0,0 @@
-
-
-Docker Container Executor
-=========================
-
-* [Overview](#Overview)
-* [Cluster Configuration](#Cluster_Configuration)
-* [Tips for connecting to a secure docker repository](#Tips_for_connecting_to_a_secure_docker_repository)
-* [Job Configuration](#Job_Configuration)
-* [Docker Image Requirements](#Docker_Image_Requirements)
-* [Working example of yarn launched docker containers](#Working_example_of_yarn_launched_docker_containers)
-
-Overview
---------
-
-[Docker](https://www.docker.io/) combines an easy-to-use interface to Linux containers with easy-to-construct image files for those containers. In short, Docker launches very light weight virtual machines.
-
-The Docker Container Executor (DCE) allows the YARN NodeManager to launch YARN containers into Docker containers. Users can specify the Docker images they want for their YARN containers. These containers provide a custom software environment in which the user's code runs, isolated from the software environment of the NodeManager. These containers can include special libraries needed by the application, and they can have different versions of Perl, Python, and even Java than what is installed on the NodeManager. Indeed, these containers can run a different flavor of Linux than what is running on the NodeManager -- although the YARN container must define all the environments and libraries needed to run the job, nothing will be shared with the NodeManager.
-
-Docker for YARN provides both consistency (all YARN containers will have the same software environment) and isolation (no interference with whatever is installed on the physical machine).
-
-Cluster Configuration
----------------------
-
-Docker Container Executor runs in non-secure mode of HDFS and YARN. It will not run in secure mode, and will exit if it detects secure mode.
-
-The DockerContainerExecutor requires Docker daemon to be running on the NodeManagers, and the Docker client installed and able to start Docker containers. To prevent timeouts while starting jobs, the Docker images to be used by a job should already be downloaded in the NodeManagers. Here's an example of how this can be done:
-
- sudo docker pull sequenceiq/hadoop-docker:2.4.1
-
-This should be done as part of the NodeManager startup.
-
-The following properties must be set in yarn-site.xml:
-
-```xml
-
- yarn.nodemanager.docker-container-executor.exec-name
- /usr/bin/docker
-
- Name or path to the Docker client. This is a required parameter. If this is empty,
- user must pass an image name as part of the job invocation(see below).
-
-
-
-
- yarn.nodemanager.container-executor.class
- org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor
-
- This is the container executor setting that ensures that all
-jobs are started with the DockerContainerExecutor.
-
-
-```
-
-Administrators should be aware that DCE doesn't currently provide user name-space isolation. This means, in particular, that software running as root in the YARN container will have root privileges in the underlying NodeManager. Put differently, DCE currently provides no better security guarantees than YARN's Default Container Executor. In fact, DockerContainerExecutor will exit if it detects secure yarn.
-
-Tips for connecting to a secure docker repository
--------------------------------------------------
-
-By default, docker images are pulled from the docker public repository. The format of a docker image url is: *username*/*image\_name*. For example, sequenceiq/hadoop-docker:2.4.1 is an image in docker public repository that contains java and hadoop.
-
-If you want your own private repository, you provide the repository url instead of your username. Therefore, the image url becomes: *private\_repo\_url*/*image\_name*. For example, if your repository is on localhost:8080, your images would be like: localhost:8080/hadoop-docker
-
-To connect to a secure docker repository, you can use the following invocation:
-
-```
- docker login [OPTIONS] [SERVER]
-
- Register or log in to a Docker registry server, if no server is specified
- "https://index.docker.io/v1/" is the default.
-
- -e, --email="" Email
- -p, --password="" Password
- -u, --username="" Username
-```
-
-If you want to login to a self-hosted registry you can specify this by adding the server name.
-
- docker login
-
-This needs to be run as part of the NodeManager startup, or as a cron job if the login session expires periodically. You can login to multiple docker repositories from the same NodeManager, but all your users will have access to all your repositories, as at present the DockerContainerExecutor does not support per-job docker login.
-
-Job Configuration
------------------
-
-Currently you cannot configure any of the Docker settings with the job configuration. You can provide Mapper, Reducer, and ApplicationMaster environment overrides for the docker images, using the following 3 JVM properties respectively(only for MR jobs):
-
-* `mapreduce.map.env`: You can override the mapper's image by passing `yarn.nodemanager.docker-container-executor.image-name`=*your_image_name* to this JVM property.
-
-* `mapreduce.reduce.env`: You can override the reducer's image by passing `yarn.nodemanager.docker-container-executor.image-name`=*your_image_name* to this JVM property.
-
-* `yarn.app.mapreduce.am.env`: You can override the ApplicationMaster's image by passing `yarn.nodemanager.docker-container-executor.image-name`=*your_image_name* to this JVM property.
-
-Docker Image Requirements
--------------------------
-
-The Docker Images used for YARN containers must meet the following requirements:
-
-The distro and version of Linux in your Docker Image can be quite different from that of your NodeManager. (Docker does have a few limitations in this regard, but you're not likely to hit them.) However, if you're using the MapReduce framework, then your image will need to be configured for running Hadoop. Java must be installed in the container, and the following environment variables must be defined in the image: JAVA_HOME, HADOOP_COMMON_PATH, HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME, HADOOP_YARN_HOME, and HADOOP_CONF_DIR
-
-Working example of yarn launched docker containers
---------------------------------------------------
-
-The following example shows how to run teragen using DockerContainerExecutor.
-
-Step 1. First ensure that YARN is properly configured with DockerContainerExecutor(see above).
-
-```xml
-
- yarn.nodemanager.docker-container-executor.exec-name
- docker -H=tcp://0.0.0.0:4243
-
- Name or path to the Docker client. The tcp socket must be
- where docker daemon is listening.
-
-
-
-
- yarn.nodemanager.container-executor.class
- org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor
-
- This is the container executor setting that ensures that all
-jobs are started with the DockerContainerExecutor.
-
-
-```
-
-Step 2. Pick a custom Docker image if you want. In this example, we'll use sequenceiq/hadoop-docker:2.4.1 from the docker hub repository. It has jdk, hadoop, and all the previously mentioned environment variables configured.
-
-Step 3. Run.
-
-```bash
-hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-${project.version}.jar \
- teragen \
- -Dmapreduce.map.env="yarn.nodemanager.docker-container-executor.image-name=sequenceiq/hadoop-docker:2.4.1" \
- -Dyarn.app.mapreduce.am.env="yarn.nodemanager.docker-container-executor.image-name=sequenceiq/hadoop-docker:2.4.1" \
- 1000 \
- teragen_out_dir
-```
-
- Once it succeeds, you can check the yarn debug logs to verify that docker indeed has launched containers.
-