Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9705

FsShell cp -p does not preserve directory attibutes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0-beta, 3.0.0-alpha1
    • Fix Version/s: 2.5.0
    • Component/s: fs
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-9338 added the -p flag to preserve file attributes when copying.

      However, cp -p does not preserve directory attributes. It'd be useful to add this functionality.

      For example, the following shows that the modified time is not preserved

      [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir /user/schu/testDir1
      [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
      Found 1 items
      drwxr-xr-x   - schu supergroup          0 2013-07-07 20:25 /user/schu/testDir1
      [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p /user/schu/testDir1 /user/schu/testDir2
      [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
      Found 2 items
      drwxr-xr-x   - schu supergroup          0 2013-07-07 20:25 /user/schu/testDir1
      drwxr-xr-x   - schu supergroup          0 2013-07-07 20:35 /user/schu/testDir2
      [schu@hdfs-snapshots-1 ~]$ 
      

      The preserve logic is in CommandWithDestination#copyFileToTarget, which is only called with files.

        protected void processPath(PathData src, PathData dst) throws IOException {
          if (src.stat.isSymlink()) {
            // TODO: remove when FileContext is supported, this needs to either                                                                                                     
            // copy the symlink or deref the symlink                                                                                                                                
            throw new PathOperationException(src.toString());
          } else if (src.stat.isFile()) {
            copyFileToTarget(src, dst);
          } else if (src.stat.isDirectory() && !isRecursive()) {
            throw new PathIsDirectoryException(src.toString());
          }
        }
      
        /**                                                                                                                                                                         
         * Copies the source file to the target.                                                                                                                                    
         * @param src item to copy                                                                                                                                                  
         * @param target where to copy the item                                                                                                                                     
         * @throws IOException if copy fails                                                                                                                                        
         */
        protected void copyFileToTarget(PathData src, PathData target) throws IOException {
          src.fs.setVerifyChecksum(verifyChecksum);
          if (src != null) {
            throw new PathExistsException("hi");
          }
          InputStream in = null;
          try {
            in = src.fs.open(src.path);
            copyStreamToTarget(in, target);
            if(preserve) {
              target.fs.setTimes(
                target.path,
                src.stat.getModificationTime(),
                src.stat.getAccessTime());
              target.fs.setOwner(
                target.path,
                src.stat.getOwner(),
                src.stat.getGroup());
              target.fs.setPermission(
                target.path,
                src.stat.getPermission());
              System.out.println("Preserving");
      
              if (src.fs.equals(target.fs)) {
                  System.out.println("Same filesystems");
                src.fs.preserveAttributes(src.path, target.path);
              }
              throw new IOException("hi");
            }
          } finally {
            IOUtils.closeStream(in);
          }
        }
      

        Attachments

        1. HADOOP-9705.2.patch
          2 kB
          Akira Ajisaka
        2. HADOOP-9705.3.patch
          3 kB
          Akira Ajisaka
        3. HADOOP-9705.4.patch
          4 kB
          Akira Ajisaka
        4. HADOOP-9705.5.patch
          14 kB
          Akira Ajisaka
        5. HADOOP-9705.patch
          3 kB
          Jean-Baptiste Onofré

          Issue Links

            Activity

              People

              • Assignee:
                aajisaka Akira Ajisaka
                Reporter:
                schu Stephen Chu
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: