Uploaded image for project: 'Sqoop (Retired)'
  1. Sqoop (Retired)
  2. SQOOP-2647

Add option for drop-if-exists when using sqoop hcat import.

Details

    Description

      To recover from a failed sqoop import via hcat using --create-hcatalog-table, there is a manual step needed to clean up the hive table i.e. (DROP TABLE IF EXISTS <table_name>). It would be great to have another flag --drop-and-create-hcatalog-table, which works like --create-hcatalog-table with an addition of drop if exists before creating the table.

      Attachments

        1. SQOOP-2647.1
          5 kB
          Muhammad Ehsan ul Haque
        2. SQOOP-2647.2
          7 kB
          Muhammad Ehsan ul Haque
        3. SQOOP-2647.3
          9 kB
          Muhammad Ehsan ul Haque

        Activity

          I would like to work on this small feature if it seems a good/valid improvement.

          ehsan Muhammad Ehsan ul Haque added a comment - I would like to work on this small feature if it seems a good/valid improvement.
          ehsan Muhammad Ehsan ul Haque added a comment - - edited

          The patch adds a parameter "--drop-and-create-hcatalog-table", which is same as "--create-hcatalog-table", with an addition that it will do a drop if exists.

          ehsan Muhammad Ehsan ul Haque added a comment - - edited The patch adds a parameter "--drop-and-create-hcatalog-table", which is same as "--create-hcatalog-table", with an addition that it will do a drop if exists.
          ehsan Muhammad Ehsan ul Haque added a comment - Review board link https://reviews.apache.org/r/41027/

          jarcec could you or anyone else you know look into this?

          ehsan Muhammad Ehsan ul Haque added a comment - jarcec could you or anyone else you know look into this?

          hshreedharan abec jarcec can anyone look into it. It would be great if it could make to 1.4.7

          ehsan Muhammad Ehsan ul Haque added a comment - hshreedharan abec jarcec can anyone look into it. It would be great if it could make to 1.4.7

          I'm happy to take a look, can you upload the patch to review board ehsan?

          jarcec Jarek Jarcec Cecho added a comment - I'm happy to take a look, can you upload the patch to review board ehsan ?

          Thanks jarcec, I have uploaded the patch to the review board as well. Here is the link on review board
          https://reviews.apache.org/r/41027/

          ehsan Muhammad Ehsan ul Haque added a comment - Thanks jarcec , I have uploaded the patch to the review board as well. Here is the link on review board https://reviews.apache.org/r/41027/

          Added some tests to validate the input arguments.

          ehsan Muhammad Ehsan ul Haque added a comment - Added some tests to validate the input arguments.

          jarcec I have added some tests now, can you look into it?
          https://reviews.apache.org/r/41027/

          ehsan Muhammad Ehsan ul Haque added a comment - jarcec I have added some tests now, can you look into it? https://reviews.apache.org/r/41027/

          I've provided tip how to properly run the HCatalog tests on the review board ehsan.

          jarcec Jarek Jarcec Cecho added a comment - I've provided tip how to properly run the HCatalog tests on the review board ehsan .

          Updated the patch with more HCAT tests.
          jarcec Please review.

          ehsan Muhammad Ehsan ul Haque added a comment - Updated the patch with more HCAT tests. jarcec Please review.

          Commit 2eecf68010df83b49eb3965a9a99de0cfbfb5828 in sqoop's branch refs/heads/trunk from jarcec
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=2eecf68 ]

          SQOOP-2647: Add option for drop-if-exists when using sqoop hcat import.

          (Muhammad Ehsan ul Haque via Jarek Jarcec Cecho)

          jira-bot ASF subversion and git services added a comment - Commit 2eecf68010df83b49eb3965a9a99de0cfbfb5828 in sqoop's branch refs/heads/trunk from jarcec [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=2eecf68 ] SQOOP-2647 : Add option for drop-if-exists when using sqoop hcat import. (Muhammad Ehsan ul Haque via Jarek Jarcec Cecho)

          Thank you for your contribution ehsan!

          Sorry that the review round took so long.

          jarcec Jarek Jarcec Cecho added a comment - Thank you for your contribution ehsan ! Sorry that the review round took so long.
          hudson Hudson added a comment -

          FAILURE: Integrated in Sqoop-hadoop100 #1005 (See https://builds.apache.org/job/Sqoop-hadoop100/1005/)
          SQOOP-2647: Add option for drop-if-exists when using sqoop hcat import. (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=2eecf68010df83b49eb3965a9a99de0cfbfb5828)

          • src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
          • src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/docs/user/hcatalog.txt
          hudson Hudson added a comment - FAILURE: Integrated in Sqoop-hadoop100 #1005 (See https://builds.apache.org/job/Sqoop-hadoop100/1005/ ) SQOOP-2647 : Add option for drop-if-exists when using sqoop hcat import. (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=2eecf68010df83b49eb3965a9a99de0cfbfb5828 ) src/test/org/apache/sqoop/hcat/HCatalogImportTest.java src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/docs/user/hcatalog.txt
          hudson Hudson added a comment -

          FAILURE: Integrated in Sqoop-hadoop200 #1045 (See https://builds.apache.org/job/Sqoop-hadoop200/1045/)
          SQOOP-2647: Add option for drop-if-exists when using sqoop hcat import. (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=2eecf68010df83b49eb3965a9a99de0cfbfb5828)

          • src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java
          • src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/docs/user/hcatalog.txt
          hudson Hudson added a comment - FAILURE: Integrated in Sqoop-hadoop200 #1045 (See https://builds.apache.org/job/Sqoop-hadoop200/1045/ ) SQOOP-2647 : Add option for drop-if-exists when using sqoop hcat import. (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=2eecf68010df83b49eb3965a9a99de0cfbfb5828 ) src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java src/test/org/apache/sqoop/hcat/HCatalogImportTest.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/docs/user/hcatalog.txt
          hudson Hudson added a comment -

          FAILURE: Integrated in Sqoop-hadoop20 #1039 (See https://builds.apache.org/job/Sqoop-hadoop20/1039/)
          SQOOP-2647: Add option for drop-if-exists when using sqoop hcat import. (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=2eecf68010df83b49eb3965a9a99de0cfbfb5828)

          • src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
          • src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java
          • src/docs/user/hcatalog.txt
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          hudson Hudson added a comment - FAILURE: Integrated in Sqoop-hadoop20 #1039 (See https://builds.apache.org/job/Sqoop-hadoop20/1039/ ) SQOOP-2647 : Add option for drop-if-exists when using sqoop hcat import. (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=2eecf68010df83b49eb3965a9a99de0cfbfb5828 ) src/test/org/apache/sqoop/hcat/HCatalogImportTest.java src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java src/docs/user/hcatalog.txt src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Sqoop-hadoop23 #1242 (See https://builds.apache.org/job/Sqoop-hadoop23/1242/)
          SQOOP-2647: Add option for drop-if-exists when using sqoop hcat import. (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=2eecf68010df83b49eb3965a9a99de0cfbfb5828)

          • src/docs/user/hcatalog.txt
          • src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java
          • src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java
          hudson Hudson added a comment - FAILURE: Integrated in Sqoop-hadoop23 #1242 (See https://builds.apache.org/job/Sqoop-hadoop23/1242/ ) SQOOP-2647 : Add option for drop-if-exists when using sqoop hcat import. (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=2eecf68010df83b49eb3965a9a99de0cfbfb5828 ) src/docs/user/hcatalog.txt src/test/org/apache/sqoop/hcat/HCatalogImportTest.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java

          People

            ehsan Muhammad Ehsan ul Haque
            ehsan Muhammad Ehsan ul Haque
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: