Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3082

archive command take wrong path for input file with current directory

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.204.1, 0.23.2, 0.23.3
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: harchive
    • Labels:
      None

      Description

      $hadoop dfs -copyFromLocal /etc/passwd .
      $hadoop dfs -lsr .
      rw------ 3 hadoopqa hdfs 6883 2011-09-23 22:37 /user/hadoopqa/passwd
      $hadoop archive -archiveName test1.har -p . passwd .
      11/09/23 22:39:22 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 4 for hadoopqa
      11/09/23 22:39:22 INFO security.TokenCache: Got dt for
      hdfs://<NN host>/user/hadoopqa/.staging/job_201109232234_0004;uri=<NN IP>:8020;t.service=<NN IP>:8020
      11/09/23 22:39:22 INFO mapred.JobClient: Running job: job_201109232234_0004
      11/09/23 22:39:23 INFO mapred.JobClient: map 0% reduce 0%
      11/09/23 22:39:34 INFO mapred.JobClient: Task Id : attempt_201109232234_0004_m_000000_0, Status : FAILED
      java.io.FileNotFoundException: File does not exist: hdfs://<NN host>/user/hadoopqa/hadoopqa/passwd
      at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:525)
      at org.apache.hadoop.tools.HadoopArchives$HArchivesMapper.map(HadoopArchives.java:697)
      at org.apache.hadoop.tools.HadoopArchives$HArchivesMapper.map(HadoopArchives.java:587)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
      at org.apache.hadoop.mapred.Child.main(Child.java:255)

      So Archiving is failing as it was finding input file at /user/hadoopqa/hadoopqa/passwd , whereas it should look for /user/hadoopqa/passwd

      1. MR-3082.patch
        3 kB
        John George
      2. MR-3082.patch
        3 kB
        John George

        Activity

        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12518251/MR-3082.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2049//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518251/MR-3082.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2049//console This message is automatically generated.
        Hide
        John George added a comment -

        The patch applies to hadoop tools directory and not to the mapreduce directory and it seems like that is the reason the patch cannot be applied cleanly. Wondering if I should create a new JIRA under common for this to work?

        Show
        John George added a comment - The patch applies to hadoop tools directory and not to the mapreduce directory and it seems like that is the reason the patch cannot be applied cleanly. Wondering if I should create a new JIRA under common for this to work?
        Hide
        Robert Joseph Evans added a comment -

        Is there a reason that you are doing

        if (!parentPath.isAbsolute()) {
            parentPath= new Path(parentPath.getFileSystem(getConf()).getWorkingDirectory(), args[i+1]);
        }
        

        instead of

        if (!parentPath.isAbsolute()) {
            parentPath= parentPath.getFileSystem(getConf()).makeQualified(parentPath);
        }
        
        Show
        Robert Joseph Evans added a comment - Is there a reason that you are doing if (!parentPath.isAbsolute()) { parentPath= new Path(parentPath.getFileSystem(getConf()).getWorkingDirectory(), args[i+1]); } instead of if (!parentPath.isAbsolute()) { parentPath= parentPath.getFileSystem(getConf()).makeQualified(parentPath); }
        Hide
        John George added a comment -

        Bobby,
        You are right - I should have done it that way. A new patch addressing your comment.

        Show
        John George added a comment - Bobby, You are right - I should have done it that way. A new patch addressing your comment.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12521201/MR-3082.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2137//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521201/MR-3082.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2137//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        I am getting test failures on that test now. Perhaps you can run it again and see if you can determine what is happening.

        Show
        Robert Joseph Evans added a comment - I am getting test failures on that test now. Perhaps you can run it again and see if you can determine what is happening.
        Hide
        John George added a comment -

        If the unit tests are run from hadoop-tools, they do fail, however, that used to be the case even before this change. I thought there was a JIRA to address that issue, but could not find it. If the unit tests are run from base directory, it passes.

        Show
        John George added a comment - If the unit tests are run from hadoop-tools, they do fail, however, that used to be the case even before this change. I thought there was a JIRA to address that issue, but could not find it. If the unit tests are run from base directory, it passes.
        Hide
        Robert Joseph Evans added a comment -

        OK You are correct about that. I ran it from the base and it looks good. Thanks +1

        Show
        Robert Joseph Evans added a comment - OK You are correct about that. I ran it from the base and it looks good. Thanks +1
        Hide
        Robert Joseph Evans added a comment -

        Thanks John, I put this into trunk, branch-2, and branch-0.23

        Show
        Robert Joseph Evans added a comment - Thanks John, I put this into trunk, branch-2, and branch-0.23
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2067 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2067/)
        MAPREDUCE-3082. Archive command take wrong path for input file with current directory (John George via bobby) (Revision 1309453)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309453
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/main/java/org/apache/hadoop/tools/HadoopArchives.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/test/java/org/apache/hadoop/tools/TestHadoopArchives.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2067 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2067/ ) MAPREDUCE-3082 . Archive command take wrong path for input file with current directory (John George via bobby) (Revision 1309453) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309453 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/main/java/org/apache/hadoop/tools/HadoopArchives.java /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/test/java/org/apache/hadoop/tools/TestHadoopArchives.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1992 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1992/)
        MAPREDUCE-3082. Archive command take wrong path for input file with current directory (John George via bobby) (Revision 1309453)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309453
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/main/java/org/apache/hadoop/tools/HadoopArchives.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/test/java/org/apache/hadoop/tools/TestHadoopArchives.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1992 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1992/ ) MAPREDUCE-3082 . Archive command take wrong path for input file with current directory (John George via bobby) (Revision 1309453) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309453 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/main/java/org/apache/hadoop/tools/HadoopArchives.java /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/test/java/org/apache/hadoop/tools/TestHadoopArchives.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2004 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2004/)
        MAPREDUCE-3082. Archive command take wrong path for input file with current directory (John George via bobby) (Revision 1309453)

        Result = ABORTED
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309453
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/main/java/org/apache/hadoop/tools/HadoopArchives.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/test/java/org/apache/hadoop/tools/TestHadoopArchives.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2004 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2004/ ) MAPREDUCE-3082 . Archive command take wrong path for input file with current directory (John George via bobby) (Revision 1309453) Result = ABORTED bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309453 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/main/java/org/apache/hadoop/tools/HadoopArchives.java /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/test/java/org/apache/hadoop/tools/TestHadoopArchives.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #219 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/219/)
        svn merge -c 1309453 from trunk FIXES: MAPREDUCE-3082. Archive command take wrong path for input file with current directory (John George via bobby) (Revision 1309455)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309455
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-archives/src/main/java/org/apache/hadoop/tools/HadoopArchives.java
        • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-archives/src/test/java/org/apache/hadoop/tools/TestHadoopArchives.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #219 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/219/ ) svn merge -c 1309453 from trunk FIXES: MAPREDUCE-3082 . Archive command take wrong path for input file with current directory (John George via bobby) (Revision 1309455) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309455 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-archives/src/main/java/org/apache/hadoop/tools/HadoopArchives.java /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-archives/src/test/java/org/apache/hadoop/tools/TestHadoopArchives.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1006 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1006/)
        MAPREDUCE-3082. Archive command take wrong path for input file with current directory (John George via bobby) (Revision 1309453)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309453
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/main/java/org/apache/hadoop/tools/HadoopArchives.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/test/java/org/apache/hadoop/tools/TestHadoopArchives.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1006 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1006/ ) MAPREDUCE-3082 . Archive command take wrong path for input file with current directory (John George via bobby) (Revision 1309453) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309453 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/main/java/org/apache/hadoop/tools/HadoopArchives.java /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/test/java/org/apache/hadoop/tools/TestHadoopArchives.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1041 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1041/)
        MAPREDUCE-3082. Archive command take wrong path for input file with current directory (John George via bobby) (Revision 1309453)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309453
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/main/java/org/apache/hadoop/tools/HadoopArchives.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/test/java/org/apache/hadoop/tools/TestHadoopArchives.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1041 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1041/ ) MAPREDUCE-3082 . Archive command take wrong path for input file with current directory (John George via bobby) (Revision 1309453) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309453 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/main/java/org/apache/hadoop/tools/HadoopArchives.java /hadoop/common/trunk/hadoop-tools/hadoop-archives/src/test/java/org/apache/hadoop/tools/TestHadoopArchives.java

          People

          • Assignee:
            John George
            Reporter:
            Rajit Saha
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development