Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: 0.20.205.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      $ hadoop distcp hftp://<NN Hostname>:50070/user/hadoopqa/input1/part-00000 /user/hadoopqa/out3
      11/09/30 18:57:59 INFO tools.DistCp: srcPaths=[hftp://<NN Hostname>:50070/user/hadoopqa/input1/part-00000]
      11/09/30 18:57:59 INFO tools.DistCp: destPath=/user/hadoopqa/out3
      11/09/30 18:58:00 INFO security.TokenCache: Got dt for
      hftp://<NN Hostname>:50070/user/hadoopqa/input1/part-00000;uri=<NN IP>:50470;t.service=<NN IP>:50470
      11/09/30 18:58:00 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 24 for hadoopqa on <NN IP>:8020
      11/09/30 18:58:00 INFO security.TokenCache: Got dt for
      /user/hadoopqa/out3;uri=<NN IP>:8020;t.service=<NN IP>:8020
      11/09/30 18:58:00 INFO tools.DistCp: /user/hadoopqa/out3 does not exist.
      11/09/30 18:58:00 INFO tools.DistCp: sourcePathsCount=1
      11/09/30 18:58:00 INFO tools.DistCp: filesToCopyCount=1
      11/09/30 18:58:00 INFO tools.DistCp: bytesToCopyCount=1.0g
      11/09/30 18:58:01 INFO mapred.JobClient: Running job: job_201109300819_0007
      11/09/30 18:58:02 INFO mapred.JobClient: map 0% reduce 0%
      11/09/30 18:58:25 INFO mapred.JobClient: Task Id : attempt_201109300819_0007_m_000000_0, Status : FAILED
      java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
      at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
      at org.apache.hadoop.mapred.Child.main(Child.java:249)

      11/09/30 18:58:41 INFO mapred.JobClient: Task Id : attempt_201109300819_0007_m_000000_1, Status : FAILED
      java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
      at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
      at org.apache.hadoop.mapred.Child.main(Child.java:249)

      11/09/30 18:58:56 INFO mapred.JobClient: Task Id : attempt_201109300819_0007_m_000000_2, Status : FAILED
      java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
      at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
      at org.apache.hadoop.mapred.Child.main(Child.java:249)

      11/09/30 18:59:14 INFO mapred.JobClient: Job complete: job_201109300819_0007
      11/09/30 18:59:14 INFO mapred.JobClient: Counters: 6
      11/09/30 18:59:14 INFO mapred.JobClient: Job Counters
      11/09/30 18:59:14 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=62380
      11/09/30 18:59:14 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
      11/09/30 18:59:14 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
      11/09/30 18:59:14 INFO mapred.JobClient: Launched map tasks=4
      11/09/30 18:59:14 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
      11/09/30 18:59:14 INFO mapred.JobClient: Failed map tasks=1
      11/09/30 18:59:14 INFO mapred.JobClient: Job Failed: # of failed Map Tasks exceeded allowed limit. FailedCount: 1.
      LastFailedTask: task_201109300819_0007_m_000000
      With failures, global counters are inaccurate; consider running with -i
      Copy failed: java.io.IOException: Job failed!
      at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1257)
      at org.apache.hadoop.tools.DistCp.copy(DistCp.java:667)
      at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)

        Issue Links

          Activity

          Rajit Saha created issue -
          Daryn Sharp made changes -
          Field Original Value New Value
          Assignee Daryn Sharp [ daryn ]
          Hide
          Rajit Saha added a comment -

          A few warnings can be useful for debugging

          From NN
          =======
          2011-09-30 18:58:11,920 WARN org.mortbay.log: /data/user/hadoopqa/input1/part-00000: java.io.IOException: Security
          enabled but user not authenticated by filter

          But the same user can do distcp over hdfs:// successfully

          Show
          Rajit Saha added a comment - A few warnings can be useful for debugging From NN ======= 2011-09-30 18:58:11,920 WARN org.mortbay.log: /data/user/hadoopqa/input1/part-00000: java.io.IOException: Security enabled but user not authenticated by filter But the same user can do distcp over hdfs:// successfully
          Hide
          Daryn Sharp added a comment -

          Yes, the selected token isn't being set and passed in the request. Agreed it's not easy to know the error w/o looking at the NN's logs.

          Show
          Daryn Sharp added a comment - Yes, the selected token isn't being set and passed in the request. Agreed it's not easy to know the error w/o looking at the NN's logs.
          Hide
          Daryn Sharp added a comment -

          Fix to call setDelegationToken with a token located in the cache.

          Move token initialization into a new method. Update initialize method to call the token init.

          Using aforementioned change to write unit tests to verify that the token is set, and that the correct token is set.

          Show
          Daryn Sharp added a comment - Fix to call setDelegationToken with a token located in the cache. Move token initialization into a new method. Update initialize method to call the token init. Using aforementioned change to write unit tests to verify that the token is set, and that the correct token is set.
          Daryn Sharp made changes -
          Attachment HDFS-2392.patch [ 12497239 ]
          Daryn Sharp made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 0.20.205.0 [ 12316392 ]
          Hide
          Daryn Sharp added a comment -

          Hftp tests pass. Didn't run full suite since nothing else stresses hftp code path.

          Show
          Daryn Sharp added a comment - Hftp tests pass. Didn't run full suite since nothing else stresses hftp code path.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12497239/HDFS-2392.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1325//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497239/HDFS-2392.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1325//console This message is automatically generated.
          Hide
          Jitendra Nath Pandey added a comment -

          +1 for the patch.

          Show
          Jitendra Nath Pandey added a comment - +1 for the patch.
          Hide
          Daryn Sharp added a comment -

          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 3 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          Show
          Daryn Sharp added a comment - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
          Hide
          Jitendra Nath Pandey added a comment -

          Committed to 205,206.

          Show
          Jitendra Nath Pandey added a comment - Committed to 205,206.
          Hide
          Jitendra Nath Pandey added a comment -

          Thanks to Daryn for the patch.

          Show
          Jitendra Nath Pandey added a comment - Thanks to Daryn for the patch.
          Hide
          Matt Foley added a comment -

          Is this also needed in trunk, or can we resolve it? Thanks.

          Show
          Matt Foley added a comment - Is this also needed in trunk, or can we resolve it? Thanks.
          Hide
          Daryn Sharp added a comment -

          I'll include it in the trunk merge of HADOOP-7510 since it was an oversight in that jira's changes.

          Show
          Daryn Sharp added a comment - I'll include it in the trunk merge of HADOOP-7510 since it was an oversight in that jira's changes.
          Matt Foley made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Hide
          Matt Foley added a comment -

          Closed upon release of 0.20.205.0

          Show
          Matt Foley added a comment - Closed upon release of 0.20.205.0
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Daryn Sharp made changes -
          Link This issue relates to HADOOP-7808 [ HADOOP-7808 ]

            People

            • Assignee:
              Daryn Sharp
              Reporter:
              Rajit Saha
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development