Hadoop Common
  1. Hadoop Common
  2. HADOOP-7361

Provide overwrite option (-overwrite/-f) in put and copyFromLocal command line options

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.1, 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      FileSystem has the API

      public void copyFromLocalFile(boolean delSrc, boolean overwrite, Path[] srcs, Path dst)

      This API provides overwrite option. But the mapping command line doesn't have this option. To maintain the consistency and better usage the command line option also can support the overwrite option like to put the files forcefully. ( put [-f] <srcpath> <dstPath>) and also for copyFromLocal command line option.

      1. HDFS-1608-test.patch
        2 kB
        Uma Maheswara Rao G
      2. HDFS-1608-src.patch
        5 kB
        Uma Maheswara Rao G
      3. HADOOP-7361-src.patch
        2 kB
        Uma Maheswara Rao G

        Issue Links

          Activity

          Uma Maheswara Rao G created issue -
          Uma Maheswara Rao G made changes -
          Field Original Value New Value
          Description FileSystem has the API

            {code:xml}


          public void copyFromLocalFile(boolean delSrc, boolean overwrite, Path[] srcs, Path dst)

          {code}
                                   
          This API provides overwrite option. But the mapping command line doesn't have this option. To maintain the consistency and better usage the command line option also can support the overwrite option like to put the files forcefully. ( put [-f] <srcpath> <dstPath>) and also for copyFromLocal command line option.
          FileSystem has the API



          *public void copyFromLocalFile(boolean delSrc, boolean overwrite, Path[] srcs, Path dst)*
               
                                   
          This API provides overwrite option. But the mapping command line doesn't have this option. To maintain the consistency and better usage the command line option also can support the overwrite option like to put the files forcefully. ( put [-f] <srcpath> <dstPath>) and also for copyFromLocal command line option.
          Hide
          Uma Maheswara Rao G added a comment -

          Fix for this defect is in FsShell.java but the file present in COMMON project. Patch need to be updated in COMMON.

          Show
          Uma Maheswara Rao G added a comment - Fix for this defect is in FsShell.java but the file present in COMMON project. Patch need to be updated in COMMON.
          Uma Maheswara Rao G made changes -
          Attachment HDFS-1608-src.patch [ 12472311 ]
          Uma Maheswara Rao G made changes -
          Attachment HDFS-1608-test.patch [ 12472312 ]
          Hide
          Uma Maheswara Rao G added a comment -

          Fix for this defect is in FsShell.java but the file present in COMMON project. Patch (HDFS-1608-src.patch) need to be updated in COMMON.

          Show
          Uma Maheswara Rao G added a comment - Fix for this defect is in FsShell.java but the file present in COMMON project. Patch ( HDFS-1608 -src.patch) need to be updated in COMMON.
          Uma Maheswara Rao G made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Harsh J added a comment -

          @OP - Thanks a ton for the patch!

          Could this issue be moved to the HADOOP project since the patch belongs there? Also, best to merge both patches into a single one.

          Show
          Harsh J added a comment - @OP - Thanks a ton for the patch! Could this issue be moved to the HADOOP project since the patch belongs there? Also, best to merge both patches into a single one.
          Todd Lipcon made changes -
          Project Hadoop HDFS [ 12310942 ] Hadoop Common [ 12310240 ]
          Key HDFS-1608 HADOOP-7361
          Affects Version/s 0.20.1 [ 12314048 ]
          Affects Version/s 0.20.2 [ 12314204 ]
          Component/s fs [ 12310689 ]
          Component/s hdfs client [ 12312928 ]
          Hide
          Todd Lipcon added a comment -

          moved the JIRA to common

          Can you please provide an up-to-date patch against trunk? thx

          Show
          Todd Lipcon added a comment - moved the JIRA to common Can you please provide an up-to-date patch against trunk? thx
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Harsh J added a comment -

          Seems to be in parallel contention with HADOOP-7176 efforts by Daryn Sharp.

          Show
          Harsh J added a comment - Seems to be in parallel contention with HADOOP-7176 efforts by Daryn Sharp.
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks to Harsh & Todd,
          I will check with trunck and provide the patch accordingly.

          Show
          Uma Maheswara Rao G added a comment - Thanks to Harsh & Todd, I will check with trunck and provide the patch accordingly.
          Hide
          Harsh J added a comment -

          Uma,

          Check out this TODO item in the new class CommandWithDestination (which encapsulates copyFromLocal and put commands amongst others). Maybe just implementing this option to be processed should now be sufficient, since I see the variable being used in the rest of the class.

          CommandWithDestination.java
            // TODO: commands should implement a -f to enable this
            protected void setOverwrite(boolean flag) {
              overwrite = flag;
            }
          
          Show
          Harsh J added a comment - Uma, Check out this TODO item in the new class CommandWithDestination (which encapsulates copyFromLocal and put commands amongst others). Maybe just implementing this option to be processed should now be sufficient, since I see the variable being used in the rest of the class. CommandWithDestination.java // TODO: commands should implement a -f to enable this protected void setOverwrite( boolean flag) { overwrite = flag; }
          Uma Maheswara Rao G made changes -
          Assignee Uma Maheswara Rao G [ umamaheswararao ]
          Uma Maheswara Rao G made changes -
          Attachment HADOOP-7361.patch [ 12485431 ]
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks Harsh for your inputs.
          Updated the patch against to latest trunk.

          Show
          Uma Maheswara Rao G added a comment - Thanks Harsh for your inputs. Updated the patch against to latest trunk.
          Uma Maheswara Rao G made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12485431/HADOOP-7361.patch
          against trunk revision 1143219.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/700//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12485431/HADOOP-7361.patch against trunk revision 1143219. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/700//console This message is automatically generated.
          Uma Maheswara Rao G made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Uma Maheswara Rao G made changes -
          Attachment HADOOP-7361.patch [ 12485431 ]
          Uma Maheswara Rao G made changes -
          Attachment HADOOP-7361-src.patch [ 12485437 ]
          Uma Maheswara Rao G made changes -
          Link This issue blocks HDFS-2131 [ HDFS-2131 ]
          Hide
          Uma Maheswara Rao G added a comment -

          Test code added as part of HDFS-2131

          Show
          Uma Maheswara Rao G added a comment - Test code added as part of HDFS-2131
          Uma Maheswara Rao G made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12485437/HADOOP-7361-src.patch
          against trunk revision 1143491.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/701//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/701//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/701//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12485437/HADOOP-7361-src.patch against trunk revision 1143491. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/701//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/701//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/701//console This message is automatically generated.
          Hide
          Uma Maheswara Rao G added a comment -

          Test code added as part of HDFS-2131

          Show
          Uma Maheswara Rao G added a comment - Test code added as part of HDFS-2131
          Hide
          Daryn Sharp added a comment -

          +1 looks good!

          Show
          Daryn Sharp added a comment - +1 looks good!
          Uma Maheswara Rao G made changes -
          Affects Version/s 0.20.1 [ 12313866 ]
          Affects Version/s 0.23.0 [ 12315569 ]
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks Daryn! for taking a look.

          Show
          Uma Maheswara Rao G added a comment - Thanks Daryn! for taking a look.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Uma!

          Also thanks Daryn for reviewing it.

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Uma! Also thanks Daryn for reviewing it.
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.23.0 [ 12315569 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #684 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/684/)
          HADOOP-7361. Provide an option, -overwrite/-f, in put and copyFromLocal shell commands. Contributed by Uma Maheswara Rao G

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1144858
          Files :

          • /hadoop/common/trunk/common/CHANGES.txt
          • /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
          • /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/CopyCommands.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #684 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/684/ ) HADOOP-7361 . Provide an option, -overwrite/-f, in put and copyFromLocal shell commands. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1144858 Files : /hadoop/common/trunk/common/CHANGES.txt /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/CommandWithDestination.java /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/CopyCommands.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #743 (See https://builds.apache.org/job/Hadoop-Common-trunk/743/)
          HADOOP-7361. Provide an option, -overwrite/-f, in put and copyFromLocal shell commands. Contributed by Uma Maheswara Rao G

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1144858
          Files :

          • /hadoop/common/trunk/common/CHANGES.txt
          • /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
          • /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/CopyCommands.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #743 (See https://builds.apache.org/job/Hadoop-Common-trunk/743/ ) HADOOP-7361 . Provide an option, -overwrite/-f, in put and copyFromLocal shell commands. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1144858 Files : /hadoop/common/trunk/common/CHANGES.txt /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/CommandWithDestination.java /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/CopyCommands.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #784 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/784/)
          HDFS-2131. Add new tests for the -overwrite/-f option in put and copyFromLocal by HADOOP-7361. Contributed by Uma Maheswara Rao G

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145843
          Files :

          • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml
          • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSShell.java
          • /hadoop/common/trunk/hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #784 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/784/ ) HDFS-2131 . Add new tests for the -overwrite/-f option in put and copyFromLocal by HADOOP-7361 . Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145843 Files : /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSShell.java /hadoop/common/trunk/hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #723 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/723/)
          HDFS-2131. Add new tests for the -overwrite/-f option in put and copyFromLocal by HADOOP-7361. Contributed by Uma Maheswara Rao G

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145843
          Files :

          • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml
          • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSShell.java
          • /hadoop/common/trunk/hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #723 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/723/ ) HDFS-2131 . Add new tests for the -overwrite/-f option in put and copyFromLocal by HADOOP-7361 . Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1145843 Files : /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSShell.java /hadoop/common/trunk/hdfs/CHANGES.txt
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          97d 7h 8m 2 Uma Maheswara Rao G 06/Jul/11 16:15
          Open Open Patch Available Patch Available
          55d 18h 12m 3 Uma Maheswara Rao G 06/Jul/11 18:01
          Patch Available Patch Available Resolved Resolved
          3d 20h 29m 1 Tsz Wo Nicholas Sze 10/Jul/11 14:31
          Resolved Resolved Closed Closed
          127d 11h 19m 1 Arun C Murthy 15/Nov/11 00:50

            People

            • Assignee:
              Uma Maheswara Rao G
              Reporter:
              Uma Maheswara Rao G
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development