Hadoop Common
  1. Hadoop Common
  2. HADOOP-7361

Provide overwrite option (-overwrite/-f) in put and copyFromLocal command line options

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.1, 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      FileSystem has the API

      public void copyFromLocalFile(boolean delSrc, boolean overwrite, Path[] srcs, Path dst)

      This API provides overwrite option. But the mapping command line doesn't have this option. To maintain the consistency and better usage the command line option also can support the overwrite option like to put the files forcefully. ( put [-f] <srcpath> <dstPath>) and also for copyFromLocal command line option.

      1. HDFS-1608-src.patch
        5 kB
        Uma Maheswara Rao G
      2. HDFS-1608-test.patch
        2 kB
        Uma Maheswara Rao G
      3. HADOOP-7361-src.patch
        2 kB
        Uma Maheswara Rao G

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #723 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/723/)
          HDFS-2131. Add new tests for the -overwrite/-f option in put and copyFromLocal by HADOOP-7361. Contributed by Uma Maheswara Rao G

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145843
          Files :

          • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml
          • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSShell.java
          • /hadoop/common/trunk/hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #723 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/723/ ) HDFS-2131 . Add new tests for the -overwrite/-f option in put and copyFromLocal by HADOOP-7361 . Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145843 Files : /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSShell.java /hadoop/common/trunk/hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #784 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/784/)
          HDFS-2131. Add new tests for the -overwrite/-f option in put and copyFromLocal by HADOOP-7361. Contributed by Uma Maheswara Rao G

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145843
          Files :

          • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml
          • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSShell.java
          • /hadoop/common/trunk/hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #784 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/784/ ) HDFS-2131 . Add new tests for the -overwrite/-f option in put and copyFromLocal by HADOOP-7361 . Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145843 Files : /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSShell.java /hadoop/common/trunk/hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #743 (See https://builds.apache.org/job/Hadoop-Common-trunk/743/)
          HADOOP-7361. Provide an option, -overwrite/-f, in put and copyFromLocal shell commands. Contributed by Uma Maheswara Rao G

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1144858
          Files :

          • /hadoop/common/trunk/common/CHANGES.txt
          • /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
          • /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/CopyCommands.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #743 (See https://builds.apache.org/job/Hadoop-Common-trunk/743/ ) HADOOP-7361 . Provide an option, -overwrite/-f, in put and copyFromLocal shell commands. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1144858 Files : /hadoop/common/trunk/common/CHANGES.txt /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/CommandWithDestination.java /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/CopyCommands.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #684 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/684/)
          HADOOP-7361. Provide an option, -overwrite/-f, in put and copyFromLocal shell commands. Contributed by Uma Maheswara Rao G

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1144858
          Files :

          • /hadoop/common/trunk/common/CHANGES.txt
          • /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
          • /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/CopyCommands.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #684 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/684/ ) HADOOP-7361 . Provide an option, -overwrite/-f, in put and copyFromLocal shell commands. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1144858 Files : /hadoop/common/trunk/common/CHANGES.txt /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/CommandWithDestination.java /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/CopyCommands.java
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Uma!

          Also thanks Daryn for reviewing it.

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Uma! Also thanks Daryn for reviewing it.
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks Daryn! for taking a look.

          Show
          Uma Maheswara Rao G added a comment - Thanks Daryn! for taking a look.
          Hide
          Daryn Sharp added a comment -

          +1 looks good!

          Show
          Daryn Sharp added a comment - +1 looks good!
          Hide
          Uma Maheswara Rao G added a comment -

          Test code added as part of HDFS-2131

          Show
          Uma Maheswara Rao G added a comment - Test code added as part of HDFS-2131
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12485437/HADOOP-7361-src.patch
          against trunk revision 1143491.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/701//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/701//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/701//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12485437/HADOOP-7361-src.patch against trunk revision 1143491. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/701//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/701//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/701//console This message is automatically generated.
          Hide
          Uma Maheswara Rao G added a comment -

          Test code added as part of HDFS-2131

          Show
          Uma Maheswara Rao G added a comment - Test code added as part of HDFS-2131
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12485431/HADOOP-7361.patch
          against trunk revision 1143219.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/700//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12485431/HADOOP-7361.patch against trunk revision 1143219. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/700//console This message is automatically generated.
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks Harsh for your inputs.
          Updated the patch against to latest trunk.

          Show
          Uma Maheswara Rao G added a comment - Thanks Harsh for your inputs. Updated the patch against to latest trunk.
          Hide
          Harsh J added a comment -

          Uma,

          Check out this TODO item in the new class CommandWithDestination (which encapsulates copyFromLocal and put commands amongst others). Maybe just implementing this option to be processed should now be sufficient, since I see the variable being used in the rest of the class.

          CommandWithDestination.java
            // TODO: commands should implement a -f to enable this
            protected void setOverwrite(boolean flag) {
              overwrite = flag;
            }
          
          Show
          Harsh J added a comment - Uma, Check out this TODO item in the new class CommandWithDestination (which encapsulates copyFromLocal and put commands amongst others). Maybe just implementing this option to be processed should now be sufficient, since I see the variable being used in the rest of the class. CommandWithDestination.java // TODO: commands should implement a -f to enable this protected void setOverwrite( boolean flag) { overwrite = flag; }
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks to Harsh & Todd,
          I will check with trunck and provide the patch accordingly.

          Show
          Uma Maheswara Rao G added a comment - Thanks to Harsh & Todd, I will check with trunck and provide the patch accordingly.
          Hide
          Harsh J added a comment -

          Seems to be in parallel contention with HADOOP-7176 efforts by Daryn Sharp.

          Show
          Harsh J added a comment - Seems to be in parallel contention with HADOOP-7176 efforts by Daryn Sharp.
          Hide
          Todd Lipcon added a comment -

          moved the JIRA to common

          Can you please provide an up-to-date patch against trunk? thx

          Show
          Todd Lipcon added a comment - moved the JIRA to common Can you please provide an up-to-date patch against trunk? thx
          Hide
          Harsh J added a comment -

          @OP - Thanks a ton for the patch!

          Could this issue be moved to the HADOOP project since the patch belongs there? Also, best to merge both patches into a single one.

          Show
          Harsh J added a comment - @OP - Thanks a ton for the patch! Could this issue be moved to the HADOOP project since the patch belongs there? Also, best to merge both patches into a single one.
          Hide
          Uma Maheswara Rao G added a comment -

          Fix for this defect is in FsShell.java but the file present in COMMON project. Patch (HDFS-1608-src.patch) need to be updated in COMMON.

          Show
          Uma Maheswara Rao G added a comment - Fix for this defect is in FsShell.java but the file present in COMMON project. Patch ( HDFS-1608 -src.patch) need to be updated in COMMON.
          Hide
          Uma Maheswara Rao G added a comment -

          Fix for this defect is in FsShell.java but the file present in COMMON project. Patch need to be updated in COMMON.

          Show
          Uma Maheswara Rao G added a comment - Fix for this defect is in FsShell.java but the file present in COMMON project. Patch need to be updated in COMMON.

            People

            • Assignee:
              Uma Maheswara Rao G
              Reporter:
              Uma Maheswara Rao G
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development