Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: nodemanager
    • Labels:
    • Flags:
      Patch

      Description

      The current implementation of DockerLinuxContainerRuntime does not follow the practice of trimming configuration values. This leads to errors if users set values containing space or newline.

      see the following YARN commits as reference:
      YARN-3395. FairScheduler: Trim whitespaces when using username for queuename.
      YARN-2869. CapacityScheduler should trim sub queue names when parse configuration.
      YARN-2843. Fixed NodeLabelsManager to trim inputs for hosts and labels so as to make them work correctly.

      and many other Hadoop/HDFS commits (just list a few):
      HDFS-9708. FSNamesystem.initAuditLoggers() doesn't trim classnames
      HDFS-2799. Trim fs.checkpoint.dir values.
      HADOOP-6578. Configuration should trim whitespace around a lot of value types
      HADOOP-6534. Trim whitespace from directory lists initializing

      Patch is available against trunk

      DockerLinuxContainerRuntime.java
      @@ -219,9 +219,9 @@ public void initialize(Configuration conf)
           dockerClient = new DockerClient(conf);
           allowedNetworks.clear();
           allowedNetworks.addAll(Arrays.asList(
      -        conf.getStrings(YarnConfiguration.NM_DOCKER_ALLOWED_CONTAINER_NETWORKS,
      +        conf.getTrimmedStrings(YarnConfiguration.NM_DOCKER_ALLOWED_CONTAINER_NETWORKS,
                   YarnConfiguration.DEFAULT_NM_DOCKER_ALLOWED_CONTAINER_NETWORKS)));
      -    defaultNetwork = conf.get(
      +    defaultNetwork = conf.getTrimmed(
               YarnConfiguration.NM_DOCKER_DEFAULT_CONTAINER_NETWORK,
               YarnConfiguration.DEFAULT_NM_DOCKER_DEFAULT_CONTAINER_NETWORK);
       
      @@ -237,7 +237,7 @@ public void initialize(Configuration conf)
             throw new ContainerExecutionException(message);
           }
       
      -    privilegedContainersAcl = new AccessControlList(conf.get(
      +    privilegedContainersAcl = new AccessControlList(conf.getTrimmed(
               YarnConfiguration.NM_DOCKER_PRIVILEGED_CONTAINERS_ACL,
               YarnConfiguration.DEFAULT_NM_DOCKER_PRIVILEGED_CONTAINERS_ACL));
         }
      @@ -439,7 +439,7 @@ public void launchContainer(ContainerRuntimeContext ctx)
               LOCALIZED_RESOURCES);
           @SuppressWarnings("unchecked")
           List<String> userLocalDirs = ctx.getExecutionAttribute(USER_LOCAL_DIRS);
      -    Set<String> capabilities = new HashSet<>(Arrays.asList(conf.getStrings(
      +    Set<String> capabilities = new HashSet<>(Arrays.asList(conf.getTrimmedStrings(
               YarnConfiguration.NM_DOCKER_CONTAINER_CAPABILITIES,
               YarnConfiguration.DEFAULT_NM_DOCKER_CONTAINER_CAPABILITIES)));
      

        Attachments

        1. YARN-5793.0000.patch
          2 kB
          Tianyin Xu
        2. YARN-5793.0001.patch
          3 kB
          Tianyin Xu

          Activity

            People

            • Assignee:
              tianyin Tianyin Xu
              Reporter:
              tianyin Tianyin Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: