[HADOOP-9705] FsShell cp -p does not preserve directory attibutes

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0-beta, 3.0.0-alpha1
• Fix Version/s: 2.5.0
• Component/s: fs
• Target Version/s: 2.5.0
• Hadoop Flags: Reviewed
• Assignee: Akira Ajisaka
• Reporter: Stephen Chu
• Created: 08/Jul/13 03:59
• Updated: 12/May/16 18:27
• Resolved: 26/Jun/14 05:41

Description

HADOOP-9338 added the -p flag to preserve file attributes when copying.

However, cp -p does not preserve directory attributes. It'd be useful to add this functionality.

For example, the following shows that the modified time is not preserved

[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir /user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
Found 1 items
drwxr-xr-x   - schu supergroup          0 2013-07-07 20:25 /user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p /user/schu/testDir1 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
Found 2 items
drwxr-xr-x   - schu supergroup          0 2013-07-07 20:25 /user/schu/testDir1
drwxr-xr-x   - schu supergroup          0 2013-07-07 20:35 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$ 

The preserve logic is in CommandWithDestination#copyFileToTarget, which is only called with files.

  protected void processPath(PathData src, PathData dst) throws IOException {
    if (src.stat.isSymlink()) {
      // TODO: remove when FileContext is supported, this needs to either                                                                                                     
      // copy the symlink or deref the symlink                                                                                                                                
      throw new PathOperationException(src.toString());
    } else if (src.stat.isFile()) {
      copyFileToTarget(src, dst);
    } else if (src.stat.isDirectory() && !isRecursive()) {
      throw new PathIsDirectoryException(src.toString());
    }
  }

  /**                                                                                                                                                                         
   * Copies the source file to the target.                                                                                                                                    
   * @param src item to copy                                                                                                                                                  
   * @param target where to copy the item                                                                                                                                     
   * @throws IOException if copy fails                                                                                                                                        
   */
  protected void copyFileToTarget(PathData src, PathData target) throws IOException {
    src.fs.setVerifyChecksum(verifyChecksum);
    if (src != null) {
      throw new PathExistsException("hi");
    }
    InputStream in = null;
    try {
      in = src.fs.open(src.path);
      copyStreamToTarget(in, target);
      if(preserve) {
        target.fs.setTimes(
          target.path,
          src.stat.getModificationTime(),
          src.stat.getAccessTime());
        target.fs.setOwner(
          target.path,
          src.stat.getOwner(),
          src.stat.getGroup());
        target.fs.setPermission(
          target.path,
          src.stat.getPermission());
        System.out.println("Preserving");

        if (src.fs.equals(target.fs)) {
            System.out.println("Same filesystems");
          src.fs.preserveAttributes(src.path, target.path);
        }
        throw new IOException("hi");
      }
    } finally {
      IOUtils.closeStream(in);
    }
  }

Issue Links

• is related to: HADOOP-10459 distcp V2 doesn't preserve root dir's attributes when -p is specified
• is related to: HADOOP-10557 FsShell -cp -pa option for preserving extended ACLs
• relates to: HADOOP-10557 FsShell -cp -pa option for preserving extended ACLs