Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3138

distcp fail copying to /user/<username>/<newtarget> (with permission on)

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.16.1
    • Fix Version/s: 0.16.4
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When distcp-ing to /user/<username>/<newtarget>, I get an error with

      Copy failed: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.fs.permission.AccessControlException: Permission denied: user=knoguchi, access=WRITE, inode="user":superuser:superusergroup:rwxr-xr-x

       
              at org.apache.hadoop.dfs.PermissionChecker.check(PermissionChecker.java:173)
              at org.apache.hadoop.dfs.PermissionChecker.check(PermissionChecker.java:154)
              at org.apache.hadoop.dfs.PermissionChecker.checkPermission(PermissionChecker.java:102)
              at org.apache.hadoop.dfs.FSNamesystem.checkPermission(FSNamesystem.java:4037)
              at org.apache.hadoop.dfs.FSNamesystem.checkAncestorAccess(FSNamesystem.java:4007)
              at org.apache.hadoop.dfs.FSNamesystem.mkdirsInternal(FSNamesystem.java:1576)
              at org.apache.hadoop.dfs.FSNamesystem.mkdirs(FSNamesystem.java:1559)
              at org.apache.hadoop.dfs.NameNode.mkdirs(NameNode.java:422)
              at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:409)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:899)
      
              at org.apache.hadoop.ipc.Client.call(Client.java:512)
              at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
              at org.apache.hadoop.dfs.$Proxy0.mkdirs(Unknown Source)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
              at org.apache.hadoop.dfs.$Proxy0.mkdirs(Unknown Source)
              at org.apache.hadoop.dfs.DFSClient.mkdirs(DFSClient.java:550)
              at org.apache.hadoop.dfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:184)
              at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:980)
              at org.apache.hadoop.util.CopyFiles.setup(CopyFiles.java:735)
              at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:525)
              at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:596)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
              at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:612)
      

      In distcp set up, we have

       
            if (!dstExists || !dstIsDir) {
              Path parent = destPath.getParent();
              dstfs.mkdirs(parent);
              logPath = new Path(parent, filename);
            }
      

      We should check if parent path exists before calling mkdir?

        Attachments

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              rangadi Raghu Angadi Assign to me
              Reporter:
              knoguchi Koji Noguchi

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment