Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-883

Remove input directory prior Sqoop import

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.2
    • Fix Version/s: 1.4.4
    • Component/s: None
    • Labels:
      None

      Description

      Sqoop will fail if target directory already exists on HDFS. This can for example happen after previously failed Sqoop import or in cases when users are re-importing the tables on regular basis.

      Right now user is forced to remove the input directory manually prior re-executing Sqoop. I think that it might be useful to add a new argument that would nuke the target directory if it exists automatically.

      I believe that this feature will be safe to use as it will be turned off by default. User will have to explicitly specify extra command line argument in order to remove any data.

      1. sqoop-883.diff
        12 kB
        Raghav Kumar Gautam
      2. SQOOP-883-2.patch
        11 kB
        Raghav Kumar Gautam
      3. SQOOP-883-3.patch
        13 kB
        Raghav Kumar Gautam

        Issue Links

          Activity

          Hide
          shrutijoshi Shruti Joshi added a comment -

          Sorry... By mistake, the status of this JIRA is changed from 'Open' to 'Patch available'...

          Show
          shrutijoshi Shruti Joshi added a comment - Sorry... By mistake, the status of this JIRA is changed from 'Open' to 'Patch available'...
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Changing the status back

          Show
          jarcec Jarek Jarcec Cecho added a comment - Changing the status back
          Hide
          raghavgautam Raghav Kumar Gautam added a comment -

          Uploading patch for the issue.

          Show
          raghavgautam Raghav Kumar Gautam added a comment - Uploading patch for the issue.
          Hide
          raghavgautam Raghav Kumar Gautam added a comment -

          Patch submitted as attachment.

          Show
          raghavgautam Raghav Kumar Gautam added a comment - Patch submitted as attachment.
          Hide
          raghavgautam Raghav Kumar Gautam added a comment -

          Uploading updated patch.

          Show
          raghavgautam Raghav Kumar Gautam added a comment - Uploading updated patch.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 5bfd84e137e8460408775e4c62bdf052bc262337 in branch refs/heads/trunk from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=5bfd84e ]

          SQOOP-883: Remove input directory prior Sqoop import

          (Raghav Kumar Gautam via Jarek Jarcec Cecho)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 5bfd84e137e8460408775e4c62bdf052bc262337 in branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=5bfd84e ] SQOOP-883 : Remove input directory prior Sqoop import (Raghav Kumar Gautam via Jarek Jarcec Cecho)
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Thank you Raghav for your contribution!

          Show
          jarcec Jarek Jarcec Cecho added a comment - Thank you Raghav for your contribution!
          Hide
          hudson Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop23 #877 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/877/)
          SQOOP-883: Remove input directory prior Sqoop import (Revision 5bfd84e137e8460408775e4c62bdf052bc262337)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5bfd84e137e8460408775e4c62bdf052bc262337
          Files :

          • src/test/com/cloudera/sqoop/mapreduce/TestImportJob.java
          • src/test/com/cloudera/sqoop/TestSqoopOptions.java
          • src/docs/user/import.txt
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/tool/ImportTool.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          Show
          hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop23 #877 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/877/ ) SQOOP-883 : Remove input directory prior Sqoop import (Revision 5bfd84e137e8460408775e4c62bdf052bc262337) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5bfd84e137e8460408775e4c62bdf052bc262337 Files : src/test/com/cloudera/sqoop/mapreduce/TestImportJob.java src/test/com/cloudera/sqoop/TestSqoopOptions.java src/docs/user/import.txt src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/tool/ImportTool.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          Hide
          hudson Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop100 #682 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/682/)
          SQOOP-883: Remove input directory prior Sqoop import (Revision 5bfd84e137e8460408775e4c62bdf052bc262337)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5bfd84e137e8460408775e4c62bdf052bc262337
          Files :

          • src/test/com/cloudera/sqoop/TestSqoopOptions.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/tool/ImportTool.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/test/com/cloudera/sqoop/mapreduce/TestImportJob.java
          • src/docs/user/import.txt
          Show
          hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop100 #682 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/682/ ) SQOOP-883 : Remove input directory prior Sqoop import (Revision 5bfd84e137e8460408775e4c62bdf052bc262337) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5bfd84e137e8460408775e4c62bdf052bc262337 Files : src/test/com/cloudera/sqoop/TestSqoopOptions.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/tool/ImportTool.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/test/com/cloudera/sqoop/mapreduce/TestImportJob.java src/docs/user/import.txt
          Hide
          hudson Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop20 #691 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/691/)
          SQOOP-883: Remove input directory prior Sqoop import (Revision 5bfd84e137e8460408775e4c62bdf052bc262337)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5bfd84e137e8460408775e4c62bdf052bc262337
          Files :

          • src/java/org/apache/sqoop/tool/ImportTool.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/test/com/cloudera/sqoop/mapreduce/TestImportJob.java
          • src/test/com/cloudera/sqoop/TestSqoopOptions.java
          • src/docs/user/import.txt
          Show
          hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop20 #691 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/691/ ) SQOOP-883 : Remove input directory prior Sqoop import (Revision 5bfd84e137e8460408775e4c62bdf052bc262337) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5bfd84e137e8460408775e4c62bdf052bc262337 Files : src/java/org/apache/sqoop/tool/ImportTool.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/test/com/cloudera/sqoop/mapreduce/TestImportJob.java src/test/com/cloudera/sqoop/TestSqoopOptions.java src/docs/user/import.txt
          Hide
          hudson Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop200 #697 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/697/)
          SQOOP-883: Remove input directory prior Sqoop import (Revision 5bfd84e137e8460408775e4c62bdf052bc262337)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5bfd84e137e8460408775e4c62bdf052bc262337
          Files :

          • src/test/com/cloudera/sqoop/TestSqoopOptions.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/docs/user/import.txt
          • src/test/com/cloudera/sqoop/mapreduce/TestImportJob.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/tool/ImportTool.java
          Show
          hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop200 #697 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/697/ ) SQOOP-883 : Remove input directory prior Sqoop import (Revision 5bfd84e137e8460408775e4c62bdf052bc262337) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5bfd84e137e8460408775e4c62bdf052bc262337 Files : src/test/com/cloudera/sqoop/TestSqoopOptions.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/docs/user/import.txt src/test/com/cloudera/sqoop/mapreduce/TestImportJob.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/tool/ImportTool.java

            People

            • Assignee:
              raghavgautam Raghav Kumar Gautam
              Reporter:
              jarcec Jarek Jarcec Cecho
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development