Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3123

Symbolic links with special chars causing container/task.sh to fail

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None

      Description

      the following job throws an exception when you have the special characters in it.

      hadoop jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* -Dmapreduce.job.queuename=queue1 -files file:///homes/user/hadoop/Streaming/data/streaming-980//InputDir#testlink!@$&*()-_+= -input Streaming/streaming-980/input.txt -mapper 'xargs cat' -reducer cat -output Streaming/streaming-980/Output -jobconf mapred.job.name=streamingTest-980 -jobconf mapreduce.job.acl-view-job=*

      Exception:
      2011-09-27 20:58:48,903 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: launchContainer:
      [container-executor, hadoopuser, 1, application_1317077272567_0239,
      container_1317077272567 0239_01_000001,
      tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_000001,
      tmp/mapred-local/nmPrivate/application_1317077272567_0239/container_1317077272567_0239_01 000001/task.sh,
      tmp/mapred-local/nmPrivate/container_1317077272567_0239_01_000001/container_1317077272567_0239_01_000001.tokens]1109221111-tests.jar:hadoop-mapreduce-p2011-09-27
      20:58:48,944 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container is : 2
      2011-09-27
      20:58:48,946 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception from container-launch :

      org.apache.hadoop.util.Shell$ExitCodeException:
      /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_000001/task.sh:
      line 26: syntax error near unexpected token `-_+='
      /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_000001/task.sh:
      line 26: `ln -sf /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir test
      ink!@$&*()-_+='kson-jaxrs-1.7.1.jar:/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar:.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar:

      at org.apache.hadoop.util.Shell.runCommand(Shell.java:261)

      at org.apache.hadoop.util.Shell.run(Shell.java:188)

      at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)

      at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:174)

      at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:197)

      at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:62)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)
      2011-09-27 20:58:48,951 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
      2011-09-27 20:58:48,951 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Processing
      container_1317077272567_0239_01_000001 of type UPDATE_DIAGNOSTICS_MSG
      2011-09-27 20:58:48,951 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
      Container exited with a non-zero exit code 2

      1. MR-3123.1.patch
        4 kB
        Hitesh Shah
      2. MR-3123.2.patch
        5 kB
        Hitesh Shah
      3. MR-3123.3.patch
        5 kB
        Hitesh Shah

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #856 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/856/)
        MAPREDUCE-3123. Fix NM to quote symlink names to escape special characters. Contributed by Hitesh Shah.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180772
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #856 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/856/ ) MAPREDUCE-3123 . Fix NM to quote symlink names to escape special characters. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180772 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #42 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/42/)
        Merge -c 1180772 from trunk to branch-0.23 to fix MAPREDUCE-3123.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180773
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #42 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/42/ ) Merge -c 1180772 from trunk to branch-0.23 to fix MAPREDUCE-3123 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180773 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #35 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/35/)
        Merge -c 1180772 from trunk to branch-0.23 to fix MAPREDUCE-3123.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180773
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #35 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/35/ ) Merge -c 1180772 from trunk to branch-0.23 to fix MAPREDUCE-3123 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180773 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #826 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/826/)
        MAPREDUCE-3123. Fix NM to quote symlink names to escape special characters. Contributed by Hitesh Shah.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180772
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #826 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/826/ ) MAPREDUCE-3123 . Fix NM to quote symlink names to escape special characters. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180772 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1069 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1069/)
        MAPREDUCE-3123. Fix NM to quote symlink names to escape special characters. Contributed by Hitesh Shah.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180772
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1069 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1069/ ) MAPREDUCE-3123 . Fix NM to quote symlink names to escape special characters. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180772 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1127 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1127/)
        MAPREDUCE-3123. Fix NM to quote symlink names to escape special characters. Contributed by Hitesh Shah.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180772
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1127 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1127/ ) MAPREDUCE-3123 . Fix NM to quote symlink names to escape special characters. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180772 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1049 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1049/)
        MAPREDUCE-3123. Fix NM to quote symlink names to escape special characters. Contributed by Hitesh Shah.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180772
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1049 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1049/ ) MAPREDUCE-3123 . Fix NM to quote symlink names to escape special characters. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180772 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
        Hide
        Arun C Murthy added a comment -

        I just committed this. Thanks Hitesh!

        Show
        Arun C Murthy added a comment - I just committed this. Thanks Hitesh!
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12498264/MR-3123.3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/974//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/974//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498264/MR-3123.3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/974//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/974//console This message is automatically generated.
        Hide
        Hitesh Shah added a comment -

        Add required apache header and minor test cleanup.

        Show
        Hitesh Shah added a comment - Add required apache header and minor test cleanup.
        Hide
        Mahadev konar added a comment -

        you forgot to add the apache header to the new file.

        Show
        Mahadev konar added a comment - you forgot to add the apache header to the new file.
        Hide
        Hitesh Shah added a comment -

        Will open a separate jira to track what should be done for escaping user input ( commands, filenames ) in general before we use/execute them.

        Show
        Hitesh Shah added a comment - Will open a separate jira to track what should be done for escaping user input ( commands, filenames ) in general before we use/execute them.
        Hide
        Hitesh Shah added a comment -

        Attaching a simple patch for now which addresses the immediate bug related to symlinks with special chars.

        Show
        Hitesh Shah added a comment - Attaching a simple patch for now which addresses the immediate bug related to symlinks with special chars.
        Hide
        Hitesh Shah added a comment -

        Vinod/Thomas/Arun - a query on task.sh - the final command i.e. "exec setsid /bin/bash -c ..." encapsulates the user provided command with "". The assumption here seems to be they can use environment variables with the command for expansion and that the onus of correctly escaping the command i.e escaping any special chars is on the users themselves. Would that be correct?

        For example, creating the symlink above is pretty simply by just quoting the file name but if the symlink is something that will be executed then special chars like "&" or even basic things like spaces can create problems if not quoted and escaped properly.

        Show
        Hitesh Shah added a comment - Vinod/Thomas/Arun - a query on task.sh - the final command i.e. "exec setsid /bin/bash -c ..." encapsulates the user provided command with "". The assumption here seems to be they can use environment variables with the command for expansion and that the onus of correctly escaping the command i.e escaping any special chars is on the users themselves. Would that be correct? For example, creating the symlink above is pretty simply by just quoting the file name but if the symlink is something that will be executed then special chars like "&" or even basic things like spaces can create problems if not quoted and escaped properly.
        Hide
        Thomas Graves added a comment -

        lines from task.sh file:

        mkdir -p jobSubmitDir
        ln -sf /tmp/mapred-local/usercache/hadoopqa/appcache/application_1317077272567_0239/filecache/-6725992875387940725/job.split jobSubmitDir/job.split
        mkdir -p jobSubmitDir
        ln -sf /tmp/mapred-local/usercache/hadoopqa/appcache/application_1317077272567_0239/filecache/-8476253876386538612/appTokens jobSubmitDir/appTokens
        mkdir -p jobSubmitDir
        ln -sf /tmp/mapred-local/usercache/hadoopqa/appcache/application_1317077272567_0239/filecache/-2497890035284830309/job.splitmetainfo jobSubmitDir/job.splitmetainfo
        ln -sf /tmp/mapred-local/usercache/hadoopqa/appcache/application_1317077272567_0239/filecache/-8047704159844243328/job.jar job.jar
        ln -sf /tmp/mapred-local/usercache/hadoopqa/appcache/application_1317077272567_0239/filecache/-6010439732820580780/job.xml job.xml
        ln sf /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir testlink!@$&*()_+=
        exec setsid /bin/bash -c "$JAVA_HOME/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.mapreduce.container.log.dir=/hadoop/var/log/mapredqa/application_1317077272567_0239/container_1317077272567_0239_01_000001 -Dyarn.app.mapreduce.container.log.filesize=0 -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1317077272567 239 1 1>/hadoop/var/log/mapredqa/application_1317077272567_0239/container_1317077272567_0239_01_000001/stdout 2>/hadoop/var/log/mapredqa/application_1317077272567_0239/container_1317077272567_0239_01_000001/stderr "

        Show
        Thomas Graves added a comment - lines from task.sh file: mkdir -p jobSubmitDir ln -sf /tmp/mapred-local/usercache/hadoopqa/appcache/application_1317077272567_0239/filecache/-6725992875387940725/job.split jobSubmitDir/job.split mkdir -p jobSubmitDir ln -sf /tmp/mapred-local/usercache/hadoopqa/appcache/application_1317077272567_0239/filecache/-8476253876386538612/appTokens jobSubmitDir/appTokens mkdir -p jobSubmitDir ln -sf /tmp/mapred-local/usercache/hadoopqa/appcache/application_1317077272567_0239/filecache/-2497890035284830309/job.splitmetainfo jobSubmitDir/job.splitmetainfo ln -sf /tmp/mapred-local/usercache/hadoopqa/appcache/application_1317077272567_0239/filecache/-8047704159844243328/job.jar job.jar ln -sf /tmp/mapred-local/usercache/hadoopqa/appcache/application_1317077272567_0239/filecache/-6010439732820580780/job.xml job.xml ln sf /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir testlink!@$&*() _+= exec setsid /bin/bash -c "$JAVA_HOME/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.mapreduce.container.log.dir=/hadoop/var/log/mapredqa/application_1317077272567_0239/container_1317077272567_0239_01_000001 -Dyarn.app.mapreduce.container.log.filesize=0 -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1317077272567 239 1 1>/hadoop/var/log/mapredqa/application_1317077272567_0239/container_1317077272567_0239_01_000001/stdout 2>/hadoop/var/log/mapredqa/application_1317077272567_0239/container_1317077272567_0239_01_000001/stderr "

          People

          • Assignee:
            Hitesh Shah
            Reporter:
            Thomas Graves
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development