Hadoop Common
  1. Hadoop Common
  2. HADOOP-8131

FsShell put doesn't correctly handle a non-existent dir

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.2
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change

      Description

      $ hadoop fs -ls
      ls: `.': No such file or directory
      $ hadoop fs -put file
      $ hadoop fs -ls
      Found 1 items
      -rw-r--r--   1 kihwal supergroup       2076 2011-11-04 10:37 .._COPYING_
      
      1. HADOOP-8131.patch
        6 kB
        Daryn Sharp
      2. HADOOP-8131.patch
        7 kB
        Daryn Sharp
      3. HADOOP-8131.patch
        7 kB
        Daryn Sharp
      4. HADOOP-8131.patch
        8 kB
        Daryn Sharp

        Issue Links

          Activity

          Hide
          Daryn Sharp added a comment -

          Koji, what is the ___ command? A copy, and you omitted the source for brevity? Does one of the variables in the target reference _temporary? I'll probably need to investigate further after your clarification. Thanks.

          Show
          Daryn Sharp added a comment - Koji, what is the ___ command? A copy, and you omitted the source for brevity? Does one of the variables in the target reference _temporary? I'll probably need to investigate further after your clarification. Thanks.
          Hide
          Koji Noguchi added a comment -

          Todd, Daryn,
          When users create their own outputs inside the streaming script, we ask them to use _temporary.
          "hadoop dfs -___ $

          {mapred_work_output_dir}

          /mycutom_path-$

          {mapred_tip_id}

          " or something like that.

          I do not want to see _temporary left out when jobs finish successfully.

          Show
          Koji Noguchi added a comment - Todd, Daryn, When users create their own outputs inside the streaming script, we ask them to use _temporary. "hadoop dfs -___ $ {mapred_work_output_dir} /mycutom_path-$ {mapred_tip_id} " or something like that. I do not want to see _temporary left out when jobs finish successfully.
          Hide
          Todd Lipcon added a comment -

          Just to confirm: you mean a real conf key, not a cmdline flag, right?

          Yep - something that could be set system-wide in core-site.xml. When users upgrade, they expect they may have to tweak some confs for the new version, but it's harder to ask them to change all of their shell scripts.

          In either case it will be a "change now or change later" scenario

          Right. The idea is that they would have some warning (a full major version) before their code stops working. Our general policy is to only make the breaking change after having the deprecated support for a full major version – in which case it would go away in 0.24.0.

          Would this bring back the issue of left out _temporary dirs? (MAPREDUCE-1272)

          I would think the MR task would be using the new non-deprecated API which doesn't recursively create parents.

          Show
          Todd Lipcon added a comment - Just to confirm: you mean a real conf key, not a cmdline flag, right? Yep - something that could be set system-wide in core-site.xml. When users upgrade, they expect they may have to tweak some confs for the new version, but it's harder to ask them to change all of their shell scripts. In either case it will be a "change now or change later" scenario Right. The idea is that they would have some warning (a full major version) before their code stops working. Our general policy is to only make the breaking change after having the deprecated support for a full major version – in which case it would go away in 0.24.0. Would this bring back the issue of left out _temporary dirs? ( MAPREDUCE-1272 ) I would think the MR task would be using the new non-deprecated API which doesn't recursively create parents.
          Hide
          Daryn Sharp added a comment -

          No, it wouldn't bring back _temporary dirs because that appears unrelated to FsShell.

          Show
          Daryn Sharp added a comment - No, it wouldn't bring back _temporary dirs because that appears unrelated to FsShell.
          Hide
          Koji Noguchi added a comment -

          I'd be in favor of a deprecated config flag which restores the old behavior

          Would this bring back the issue of left out _temporary dirs? (MAPREDUCE-1272)

          Show
          Koji Noguchi added a comment - I'd be in favor of a deprecated config flag which restores the old behavior Would this bring back the issue of left out _temporary dirs? ( MAPREDUCE-1272 )
          Hide
          Daryn Sharp added a comment -

          Just to confirm: you mean a real conf key, not a cmdline flag, right? In either case it will be a "change now or change later" scenario. I'm curious when would you anticipate a good time to remove a new & deprecated conf? Would it live all the way through 23? As Eli pointed out, an alpha may be the best time to remove the behavior. Thoughts?

          Show
          Daryn Sharp added a comment - Just to confirm: you mean a real conf key, not a cmdline flag, right? In either case it will be a "change now or change later" scenario. I'm curious when would you anticipate a good time to remove a new & deprecated conf? Would it live all the way through 23? As Eli pointed out, an alpha may be the best time to remove the behavior. Thoughts?
          Hide
          Todd Lipcon added a comment -

          If it's not too huge a pain, I'd be in favor of a deprecated config flag which restores the old behavior (while emitting a warning that it's deprecated and to be removed in a future version). This will help people migrate to 0.23, since I'm sure there are lots of cases where people have shell scripts running as part of production workflows.

          Show
          Todd Lipcon added a comment - If it's not too huge a pain, I'd be in favor of a deprecated config flag which restores the old behavior (while emitting a warning that it's deprecated and to be removed in a future version). This will help people migrate to 0.23, since I'm sure there are lots of cases where people have shell scripts running as part of production workflows.
          Hide
          Eli Collins added a comment -

          I like the new behavior as well, and I suppose an alpha is the best time to introduce it. How about updating this jira with a release note then? We'll file a Pig to update the tests to work against 23.

          Show
          Eli Collins added a comment - I like the new behavior as well, and I suppose an alpha is the best time to introduce it. How about updating this jira with a release note then? We'll file a Pig to update the tests to work against 23.
          Hide
          Daryn Sharp added a comment -

          @Eli: I believe the auto-create was removed with the earlier redesign but crept back in later so I didn't view this particular jira as incompatible. Since the goal has been to mimic the real shell, this non-posix "feature" was intentionally (again?) removed to solve this jira.

          It also resolves the inconsistency between single source vs. multiple sources copies. Single source copy would formerly auto-create non-existent dirs, but a multi-source copy fails if the target isn't an existing dir. Ergo, copying a glob to a non-existent path was indeterminate depending on the number of glob matches: 1 match auto-created, more than 1 match failed.

          Internally, we found the impact to be minimal (a single mkdir), so we favored retaining the posix semantics of copy. Some relied on the auto-create because mkdir failed if the directory already exists, thus HADOOP-8175 added a -p flag.

          If you feel strongly on restoring the former non-posix behavior and single/multi source inconsistency, I can look into doing so.

          Show
          Daryn Sharp added a comment - @Eli: I believe the auto-create was removed with the earlier redesign but crept back in later so I didn't view this particular jira as incompatible. Since the goal has been to mimic the real shell, this non-posix "feature" was intentionally (again?) removed to solve this jira. It also resolves the inconsistency between single source vs. multiple sources copies. Single source copy would formerly auto-create non-existent dirs, but a multi-source copy fails if the target isn't an existing dir. Ergo, copying a glob to a non-existent path was indeterminate depending on the number of glob matches: 1 match auto-created, more than 1 match failed. Internally, we found the impact to be minimal (a single mkdir), so we favored retaining the posix semantics of copy. Some relied on the auto-create because mkdir failed if the directory already exists, thus HADOOP-8175 added a -p flag. If you feel strongly on restoring the former non-posix behavior and single/multi source inconsistency, I can look into doing so.
          Hide
          Eli Collins added a comment -

          Hey Daryn,

          Should we restore the old behavior for 23? This is going to break people, eg scripts that
          do hadoop fs -put file1 dir1/file1 w/o dir1 already created used to work and won't anymore.

          Thanks,
          Eli

          Show
          Eli Collins added a comment - Hey Daryn, Should we restore the old behavior for 23? This is going to break people, eg scripts that do hadoop fs -put file1 dir1/file1 w/o dir1 already created used to work and won't anymore. Thanks, Eli
          Hide
          Bilung Lee added a comment -

          Mark this as an incompatible change as the code throws an exception now but it doesn't previously (when parent directory doesn't exist). The unit tests and the e2e tests of Pig will be broken after using a Hadoop with this patch.

          Show
          Bilung Lee added a comment - Mark this as an incompatible change as the code throws an exception now but it doesn't previously (when parent directory doesn't exist). The unit tests and the e2e tests of Pig will be broken after using a Hadoop with this patch.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1008 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1008/)
          HADOOP-8131. FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296386)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296386
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1008 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1008/ ) HADOOP-8131 . FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296386) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296386 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #214 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/214/)
          svn merge -c 1296386 trunk to branch-0.23 FIXES HADOOP-8131. FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296389)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296389
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #214 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/214/ ) svn merge -c 1296386 trunk to branch-0.23 FIXES HADOOP-8131 . FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296389) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296389 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #973 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/973/)
          HADOOP-8131. FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296386)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296386
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #973 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/973/ ) HADOOP-8131 . FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296386) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296386 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #186 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/186/)
          svn merge -c 1296386 trunk to branch-0.23 FIXES HADOOP-8131. FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296389)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296389
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #186 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/186/ ) svn merge -c 1296386 trunk to branch-0.23 FIXES HADOOP-8131 . FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296389) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296389 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #628 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/628/)
          svn merge -c 1296386 trunk to branch-0.23 FIXES HADOOP-8131. FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296389)

          Result = ABORTED
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296389
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #628 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/628/ ) svn merge -c 1296386 trunk to branch-0.23 FIXES HADOOP-8131 . FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296389) Result = ABORTED bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296389 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1828 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1828/)
          HADOOP-8131. FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296386)

          Result = ABORTED
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296386
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1828 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1828/ ) HADOOP-8131 . FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296386) Result = ABORTED bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296386 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #629 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/629/)
          svn merge -c 1296386 trunk to branch-0.23 FIXES HADOOP-8131. FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296389)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296389
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #629 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/629/ ) svn merge -c 1296386 trunk to branch-0.23 FIXES HADOOP-8131 . FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296389) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296389 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #618 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/618/)
          svn merge -c 1296386 trunk to branch-0.23 FIXES HADOOP-8131. FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296389)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296389
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #618 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/618/ ) svn merge -c 1296386 trunk to branch-0.23 FIXES HADOOP-8131 . FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296389) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296389 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1821 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1821/)
          HADOOP-8131. FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296386)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296386
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1821 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1821/ ) HADOOP-8131 . FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296386) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296386 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1895 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1895/)
          HADOOP-8131. FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296386)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296386
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1895 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1895/ ) HADOOP-8131 . FsShell put doesn't correctly handle a non-existent dir (Daryn Sharp via bobby) (Revision 1296386) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296386 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/PathData.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellReturnCode.java
          Hide
          Robert Joseph Evans added a comment -

          Thanks Daryn, I just commited this to trunk, branch-0.23 and branch-0.23.2

          Show
          Robert Joseph Evans added a comment - Thanks Daryn, I just commited this to trunk, branch-0.23 and branch-0.23.2
          Hide
          Robert Joseph Evans added a comment -

          Jenkins gave it a +1 and the code looks good to me +1.

          Show
          Robert Joseph Evans added a comment - Jenkins gave it a +1 and the code looks good to me +1.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12516856/HADOOP-8131.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/659//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/659//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516856/HADOOP-8131.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/659//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/659//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          Fixed the other test to refer to directories in the test build root so it doesn't pass or fail based on whether /test exists.

          Show
          Daryn Sharp added a comment - Fixed the other test to refer to directories in the test build root so it doesn't pass or fail based on whether /test exists.
          Hide
          Daryn Sharp added a comment -

          Test is referring to path outside build.test.root – /test. I have a /test directory... Will fix the test.

          Show
          Daryn Sharp added a comment - Test is referring to path outside build.test.root – /test. I have a /test directory... Will fix the test.
          Hide
          Robert Joseph Evans added a comment -

          Oh wait so did Jenkins

          Show
          Robert Joseph Evans added a comment - Oh wait so did Jenkins
          Hide
          Robert Joseph Evans added a comment -

          I am getting a test failure, and Jenkins has not yet reported back its results.

          Running org.apache.hadoop.fs.TestFsShellReturnCode
          Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.115 sec <<< FAILURE!

          I get the same failure each time.

          Show
          Robert Joseph Evans added a comment - I am getting a test failure, and Jenkins has not yet reported back its results. Running org.apache.hadoop.fs.TestFsShellReturnCode Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.115 sec <<< FAILURE! I get the same failure each time.
          Hide
          Daryn Sharp added a comment -

          Test is passing for me!! It appears to be referring to paths in /tmp, will see what happens when I create those files as debris.

          Show
          Daryn Sharp added a comment - Test is passing for me!! It appears to be referring to paths in /tmp, will see what happens when I create those files as debris.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12516848/HADOOP-8131.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.fs.TestFsShellReturnCode

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/658//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/658//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516848/HADOOP-8131.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.TestFsShellReturnCode +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/658//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/658//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12516846/HADOOP-8131.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.fs.viewfs.TestViewFsTrash
          org.apache.hadoop.fs.TestFsShellReturnCode

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/657//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/657//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516846/HADOOP-8131.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.viewfs.TestViewFsTrash org.apache.hadoop.fs.TestFsShellReturnCode +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/657//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/657//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          Removed unnecessary commented code in the test. FsShell dumps to stdout/stderr so I had to print to stdout to properly interlace the output. That's also what other FsShell tests do.

          The equals stuff is checking the basename of the path to catch ., .., dir/, dir/., dir/.. since Path will eat the chars off when it normalizes and/or fully qualifies. That makes it "look" like a non-existent file. The shell tracks the exact path supplied by the user, so it checks that basename for "it's a dir" conditions.

          Show
          Daryn Sharp added a comment - Removed unnecessary commented code in the test. FsShell dumps to stdout/stderr so I had to print to stdout to properly interlace the output. That's also what other FsShell tests do. The equals stuff is checking the basename of the path to catch ., .., dir/, dir/., dir/.. since Path will eat the chars off when it normalizes and/or fully qualifies. That makes it "look" like a non-existent file. The shell tracks the exact path supplied by the user, so it checks that basename for "it's a dir" conditions.
          Hide
          Robert Joseph Evans added a comment -

          From a quick look at the code it mostly looks good. I don't like the name.equals in the checking for a parent, but like the comment says there really is no other way I can think of to know if it is a directory or not, especially a directory that does not exist.

          I also noticed some commented out code in the tests, and some calls to system.out. Please remove the commented out code, and verify that call of the calls to System.out are what you want. It might be better to change them to be log statements.

          I will look at the logic more closely now.

          Show
          Robert Joseph Evans added a comment - From a quick look at the code it mostly looks good. I don't like the name.equals in the checking for a parent, but like the comment says there really is no other way I can think of to know if it is a directory or not, especially a directory that does not exist. I also noticed some commented out code in the tests, and some calls to system.out. Please remove the commented out code, and verify that call of the calls to System.out are what you want. It might be better to change them to be log statements. I will look at the logic more closely now.
          Hide
          Daryn Sharp added a comment -

          wrong patch, missing file

          Show
          Daryn Sharp added a comment - wrong patch, missing file
          Hide
          Daryn Sharp added a comment -

          Made behavior consistent with other arg handling of put by throwing FNF if parent dir of a copy doesn't exists.

          Show
          Daryn Sharp added a comment - Made behavior consistent with other arg handling of put by throwing FNF if parent dir of a copy doesn't exists.
          Hide
          Daryn Sharp added a comment -

          When using a 1-arg put, the shell default to using the pwd. If the target doesn't exist, the shell assumes it's a rename of the src file. So the temp file "$p.COPYING" becomes "..COPYING". The move at the end of the copy tries to rename the temp file to ".", which is interpreted as the pwd, so the rename is a no-op.

          Show
          Daryn Sharp added a comment - When using a 1-arg put, the shell default to using the pwd. If the target doesn't exist, the shell assumes it's a rename of the src file. So the temp file "$p. COPYING " becomes ".. COPYING ". The move at the end of the copy tries to rename the temp file to ".", which is interpreted as the pwd, so the rename is a no-op.

            People

            • Assignee:
              Daryn Sharp
              Reporter:
              Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development