Hadoop Common
  1. Hadoop Common
  2. HADOOP-8551

fs -mkdir creates parent directories without the -p option

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.0-alpha, 3.0.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: fs
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      FsShell's "mkdir" no longer implicitly creates all non-existent parent directories. The command adopts the posix compliant behavior of requiring the "-p" flag to auto-create parent directories.

      Description

      hadoop fs -mkdir foo/bar will work even if bar is not present. It should only work if -p is given and foo is not present.

      1. HADOOP-8551.patch
        46 kB
        Robert Joseph Evans
      2. HADOOP-8551.patch
        46 kB
        John George
      3. HADOOP-8551.patch
        45 kB
        John George
      4. HADOOP-8551.patch
        1 kB
        Daryn Sharp

        Issue Links

          Activity

          Hide
          Daryn Sharp added a comment -

          This should fix it, but need to write tests.

          Show
          Daryn Sharp added a comment - This should fix it, but need to write tests.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537196/HADOOP-8551.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1207//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1207//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537196/HADOOP-8551.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1207//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1207//console This message is automatically generated.
          Hide
          Robert Joseph Evans added a comment -

          The change looks good to me +1. I'll check it in.

          Show
          Robert Joseph Evans added a comment - The change looks good to me +1. I'll check it in.
          Hide
          Robert Joseph Evans added a comment -

          Thanks John and Daryn. I put this into trunk, branch-2, and branch-0.23

          Show
          Robert Joseph Evans added a comment - Thanks John and Daryn. I put this into trunk, branch-2, and branch-0.23
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2564 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2564/)
          HADOOP-8551. fs -mkdir creates parent directories without the -p option (Daryn Sharp and John George via bobby) (Revision 1363435)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363435
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2564 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2564/ ) HADOOP-8551 . fs -mkdir creates parent directories without the -p option (Daryn Sharp and John George via bobby) (Revision 1363435) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363435 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2499 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2499/)
          HADOOP-8551. fs -mkdir creates parent directories without the -p option (Daryn Sharp and John George via bobby) (Revision 1363435)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363435
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2499 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2499/ ) HADOOP-8551 . fs -mkdir creates parent directories without the -p option (Daryn Sharp and John George via bobby) (Revision 1363435) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363435 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2521 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2521/)
          HADOOP-8551. fs -mkdir creates parent directories without the -p option (Daryn Sharp and John George via bobby) (Revision 1363435)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363435
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2521 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2521/ ) HADOOP-8551 . fs -mkdir creates parent directories without the -p option (Daryn Sharp and John George via bobby) (Revision 1363435) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363435 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Robert Joseph Evans added a comment -

          -mkdir a
          -mkdir a/b/ (Fails)
          -mkdir a/b (Succeeds)

          I am going to revert this until this is fixed, thanks for catching this John.

          Show
          Robert Joseph Evans added a comment - -mkdir a -mkdir a/b/ (Fails) -mkdir a/b (Succeeds) I am going to revert this until this is fixed, thanks for catching this John.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2568 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2568/)
          Reverting HADOOP-8551. fs -mkdir creates parent directories without the -p option (Revision 1363547)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363547
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2568 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2568/ ) Reverting HADOOP-8551 . fs -mkdir creates parent directories without the -p option (Revision 1363547) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363547 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2503 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2503/)
          Reverting HADOOP-8551. fs -mkdir creates parent directories without the -p option (Revision 1363547)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363547
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2503 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2503/ ) Reverting HADOOP-8551 . fs -mkdir creates parent directories without the -p option (Revision 1363547) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363547 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2524 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2524/)
          Reverting HADOOP-8551. fs -mkdir creates parent directories without the -p option (Revision 1363547)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363547
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2524 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2524/ ) Reverting HADOOP-8551 . fs -mkdir creates parent directories without the -p option (Revision 1363547) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363547 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #319 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/319/)
          Reverting HADOOP-8551. fs -mkdir creates parent directories without the -p option (Revision 1363550)
          svn merge -c 1363435 FIXES: HADOOP-8551. fs -mkdir creates parent directories without the -p option (Daryn Sharp and John George via bobby) (Revision 1363438)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363550
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml

          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363438
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #319 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/319/ ) Reverting HADOOP-8551 . fs -mkdir creates parent directories without the -p option (Revision 1363550) svn merge -c 1363435 FIXES: HADOOP-8551 . fs -mkdir creates parent directories without the -p option (Daryn Sharp and John George via bobby) (Revision 1363438) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363550 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363438 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1142 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1142/)
          Reverting HADOOP-8551. fs -mkdir creates parent directories without the -p option (Revision 1363547)
          HADOOP-8551. fs -mkdir creates parent directories without the -p option (Daryn Sharp and John George via bobby) (Revision 1363435)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363547
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml

          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363435
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1142 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1142/ ) Reverting HADOOP-8551 . fs -mkdir creates parent directories without the -p option (Revision 1363547) HADOOP-8551 . fs -mkdir creates parent directories without the -p option (Daryn Sharp and John George via bobby) (Revision 1363435) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363547 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363435 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          John George added a comment -

          Attaching a new patch with the fix. Also added a new test to verify mkdir a/b/ works when dir 'a' exists.

          Show
          John George added a comment - Attaching a new patch with the fix. Also added a new test to verify mkdir a/b/ works when dir 'a' exists.
          Hide
          Robert Joseph Evans added a comment -

          Kicking Jenkins

          Show
          Robert Joseph Evans added a comment - Kicking Jenkins
          Hide
          Robert Joseph Evans added a comment -

          The patch looks good to me. It is a little ugly, but I cannot think of a better way to strip off the trailing /. +1 pending Jenkins, which still has not come back. I'll try and kick it again.

          Show
          Robert Joseph Evans added a comment - The patch looks good to me. It is a little ugly, but I cannot think of a better way to strip off the trailing /. +1 pending Jenkins, which still has not come back. I'll try and kick it again.
          Hide
          Robert Joseph Evans added a comment -

          John, The Jenkins build for this keeps getting aborted. Could you run test-patch yourself?

          Show
          Robert Joseph Evans added a comment - John, The Jenkins build for this keeps getting aborted. Could you run test-patch yourself?
          Hide
          John George added a comment -

          -1 overall.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 41 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 16 warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 11 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          Show
          John George added a comment - -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 41 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 16 warning messages. -1 javac. The patch appears to cause tar ant target to fail. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 11 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537732/HADOOP-8551.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.ha.TestZKFailoverController
          org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
          org.apache.hadoop.hdfs.TestPersistBlocks

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1214//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1214//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537732/HADOOP-8551.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.ha.TestZKFailoverController org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl org.apache.hadoop.hdfs.TestPersistBlocks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1214//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1214//console This message is automatically generated.
          Hide
          John George added a comment -

          The test failures are unrelated. They seem to be failing in other builds as well

          Show
          John George added a comment - The test failures are unrelated. They seem to be failing in other builds as well
          Hide
          Robert Joseph Evans added a comment -

          Yes I have seen them failing in other builds +1. I'll check this in.

          Show
          Robert Joseph Evans added a comment - Yes I have seen them failing in other builds +1. I'll check this in.
          Hide
          Robert Joseph Evans added a comment -

          Thanks John,

          I put this into trunk, branch-2, and branch-0.23 again.

          Show
          Robert Joseph Evans added a comment - Thanks John, I put this into trunk, branch-2, and branch-0.23 again.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2584 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2584/)
          HADOOP-8551. fs -mkdir creates parent directories without the -p option (John George via bobby) (Revision 1365588)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1365588
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2584 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2584/ ) HADOOP-8551 . fs -mkdir creates parent directories without the -p option (John George via bobby) (Revision 1365588) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1365588 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2519 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2519/)
          HADOOP-8551. fs -mkdir creates parent directories without the -p option (John George via bobby) (Revision 1365588)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1365588
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2519 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2519/ ) HADOOP-8551 . fs -mkdir creates parent directories without the -p option (John George via bobby) (Revision 1365588) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1365588 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #325 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/325/)
          svn merge -c 1365588 FIXES: HADOOP-8551. fs -mkdir creates parent directories without the -p option (John George via bobby) (Revision 1365590)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1365590
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #325 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/325/ ) svn merge -c 1365588 FIXES: HADOOP-8551 . fs -mkdir creates parent directories without the -p option (John George via bobby) (Revision 1365590) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1365590 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1116 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1116/)
          HADOOP-8551. fs -mkdir creates parent directories without the -p option (John George via bobby) (Revision 1365588)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1365588
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1116 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1116/ ) HADOOP-8551 . fs -mkdir creates parent directories without the -p option (John George via bobby) (Revision 1365588) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1365588 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1148 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1148/)
          HADOOP-8551. fs -mkdir creates parent directories without the -p option (John George via bobby) (Revision 1365588)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1365588
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1148 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1148/ ) HADOOP-8551 . fs -mkdir creates parent directories without the -p option (John George via bobby) (Revision 1365588) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1365588 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          Hide
          Suresh Srinivas added a comment -

          This is an incompatible change right?

          Show
          Suresh Srinivas added a comment - This is an incompatible change right?
          Hide
          Daryn Sharp added a comment -

          Yes.

          Show
          Daryn Sharp added a comment - Yes.
          Hide
          Suresh Srinivas added a comment - - edited

          We should update the release notes to indicate that this is an incompatible change. Please add in release notes details about incompatibility and how to work around it. Also we need to move the change description of this jira to incompatible section in CHANGES.txt.

          Does this not cause issues at Yahoo?

          Show
          Suresh Srinivas added a comment - - edited We should update the release notes to indicate that this is an incompatible change. Please add in release notes details about incompatibility and how to work around it. Also we need to move the change description of this jira to incompatible section in CHANGES.txt. Does this not cause issues at Yahoo?
          Hide
          Suresh Srinivas added a comment -

          Checked the Incompatible change checkbox.

          Show
          Suresh Srinivas added a comment - Checked the Incompatible change checkbox.
          Hide
          Daryn Sharp added a comment -

          No, this doesn't affect yahoo. Our customers have already updated their jobs that require multi-level directory creation.

          Show
          Daryn Sharp added a comment - No, this doesn't affect yahoo. Our customers have already updated their jobs that require multi-level directory creation.
          Hide
          Suresh Srinivas added a comment -

          Also we need to move the change description of this jira to incompatible section in CHANGES.txt.

          Can you please take care of this as well for all the relevant releases?

          Show
          Suresh Srinivas added a comment - Also we need to move the change description of this jira to incompatible section in CHANGES.txt. Can you please take care of this as well for all the relevant releases?
          Hide
          Eli Collins added a comment -

          I think it's actually worth considering reverting this from branch-2 and having a warning there about it being deprecated in 2.x. I reverted this change from CDH because it broke some many internal users and tests.

          Show
          Eli Collins added a comment - I think it's actually worth considering reverting this from branch-2 and having a warning there about it being deprecated in 2.x. I reverted this change from CDH because it broke some many internal users and tests.
          Hide
          Daryn Sharp added a comment -

          By adopting better posix compliance, there are a number of other "incompatible" changes in the shell. Such as nothing auto-magically creates parent dirs anymore. It was an oversight that mkdir did since the -p flag was there. It seems like a minor change for users to add "-p", but that's just my $.02.

          Show
          Daryn Sharp added a comment - By adopting better posix compliance, there are a number of other "incompatible" changes in the shell. Such as nothing auto-magically creates parent dirs anymore. It was an oversight that mkdir did since the -p flag was there. It seems like a minor change for users to add "-p", but that's just my $.02.
          Hide
          Eli Collins added a comment -

          That's what I thought as well, however w/in a couple week of pulling it in there were enough tests/programs failing and people complaining that I reverted it. Apparently a lot of people depend on this behavior. People have been fine with all the other incompatible shell changes, have left all those in.

          Show
          Eli Collins added a comment - That's what I thought as well, however w/in a couple week of pulling it in there were enough tests/programs failing and people complaining that I reverted it. Apparently a lot of people depend on this behavior. People have been fine with all the other incompatible shell changes, have left all those in.
          Hide
          Robert Joseph Evans added a comment -

          If the community feels that it should be reverted in branch-2 then I am fine with reverting it in branch-0.23 as well. We just want to be sure that there is a deprecation warning, and preferably that it only prints out when a -p would be required in the future, but was not given.

          Show
          Robert Joseph Evans added a comment - If the community feels that it should be reverted in branch-2 then I am fine with reverting it in branch-0.23 as well. We just want to be sure that there is a deprecation warning, and preferably that it only prints out when a -p would be required in the future, but was not given.
          Hide
          Daryn Sharp added a comment -

          If it's too much of a problem to s/mkdir /mkdir -p /, rather than reverting I'd suggest a config option (I know, I know) that in generally can be used to enable/disable deprecated behavior in the shell.

          Show
          Daryn Sharp added a comment - If it's too much of a problem to s/mkdir /mkdir -p /, rather than reverting I'd suggest a config option (I know, I know) that in generally can be used to enable/disable deprecated behavior in the shell.
          Hide
          Robert Joseph Evans added a comment -

          please no new config. I know you don't want to revert this, but a config is not going to make it any easier to get people off of the old behavior, it will just give them a crutch they they will start to rely on.

          Show
          Robert Joseph Evans added a comment - please no new config. I know you don't want to revert this, but a config is not going to make it any easier to get people off of the old behavior, it will just give them a crutch they they will start to rely on.
          Hide
          Daryn Sharp added a comment -

          We'll disable and mark it final so they can't use deprecated behavior. Cloudera won't.

          Show
          Daryn Sharp added a comment - We'll disable and mark it final so they can't use deprecated behavior. Cloudera won't.
          Hide
          Eli Collins added a comment -

          If it's too much of a problem to s/mkdir /mkdir -p /

          Just replacing mkdir with "mkdir -p" doesn't fix things because Hadoop 1.x doesn't support the "-p" option so if you just add "-p" everywhere your programs break on old releases. I'm surprised you haven't hit this trying to deploy 0.23.

          Show
          Eli Collins added a comment - If it's too much of a problem to s/mkdir /mkdir -p / Just replacing mkdir with "mkdir -p" doesn't fix things because Hadoop 1.x doesn't support the "-p" option so if you just add "-p" everywhere your programs break on old releases. I'm surprised you haven't hit this trying to deploy 0.23.
          Hide
          Daryn Sharp added a comment -

          Our customers all updated their jobs.

          Show
          Daryn Sharp added a comment - Our customers all updated their jobs.
          Hide
          Eli Collins added a comment -

          And I guess those jobs will never run on older 205 clusters. We have to worry about jobs (and tests) that run on multiple versions and didn't want to require people update their jobs (most of the other shell changes that were incompatible didn't actually require people update their jobs).

          Show
          Eli Collins added a comment - And I guess those jobs will never run on older 205 clusters. We have to worry about jobs (and tests) that run on multiple versions and didn't want to require people update their jobs (most of the other shell changes that were incompatible didn't actually require people update their jobs).
          Hide
          Arun C Murthy added a comment -

          A major boundary (2.x) seems like the right time to 'fix' this.

          Eli - we could add 'mkdir -p' to branch-1 too.

          Show
          Arun C Murthy added a comment - A major boundary (2.x) seems like the right time to 'fix' this. Eli - we could add 'mkdir -p' to branch-1 too.
          Hide
          Eli Collins added a comment -

          That's a good idea, filed HADOOP-8873.

          Even with this change it's still somewhat of an issue because while the command will now work on both a 1.x and 2.x release, it will still fail on the bulk of the 1.x install base.

          Show
          Eli Collins added a comment - That's a good idea, filed HADOOP-8873 . Even with this change it's still somewhat of an issue because while the command will now work on both a 1.x and 2.x release, it will still fail on the bulk of the 1.x install base.

            People

            • Assignee:
              John George
              Reporter:
              Robert Joseph Evans
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development