Hive
  1. Hive
  2. HIVE-2602

add support for insert partition overwrite(...) if not exists

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      INSERT OVERWRITE TABLE X PARTITION (a=b, c=d) IF NOT EXISTS ...

      The partition should be created and written if and only if it's not there already.

      The support can be added for dynamic partitions in the future, but this jira is for adding this support for static partitions.

      1. ASF.LICENSE.NOT.GRANTED--HIVE-2602.D579.1.patch
        25 kB
        Phabricator
      2. ASF.LICENSE.NOT.GRANTED--HIVE-2602.D579.2.patch
        26 kB
        Phabricator
      3. ASF.LICENSE.NOT.GRANTED--HIVE-2602.D879.1.patch
        33 kB
        Phabricator
      4. HIVE-2602.1.patch
        33 kB
        Chinna Rao Lalam
      5. HIVE-2602.2.patch
        33 kB
        Chinna Rao Lalam
      6. HIVE-2602.3.patch
        26 kB
        Chinna Rao Lalam
      7. HIVE-2602.patch
        26 kB
        Chinna Rao Lalam

        Activity

        Hide
        Chinna Rao Lalam added a comment -

        I am interested to work on this just i am conforming the requirement...

        If INSERT OVERWRITE TABLE X PARTITION (a=b, c=d)... earlier it is working like
        If partition not exists it will create the partition and add data into it.
        If partition exists it will delete the old data and it will add new data.

        After change INSERT OVERWRITE TABLE X PARTITION (a=b, c=d) IF NOT EXISTS ...
        If partition not exists it will create the partition and add data into it.
        If partition exists it wont do any thing (it wont delete old data and wont add new data)
        and this should happen at the time of query compilation like
        "create table if not exists as select"

        Show
        Chinna Rao Lalam added a comment - I am interested to work on this just i am conforming the requirement... If INSERT OVERWRITE TABLE X PARTITION (a=b, c=d)... earlier it is working like If partition not exists it will create the partition and add data into it. If partition exists it will delete the old data and it will add new data. After change INSERT OVERWRITE TABLE X PARTITION (a=b, c=d) IF NOT EXISTS ... If partition not exists it will create the partition and add data into it. If partition exists it wont do any thing (it wont delete old data and wont add new data) and this should happen at the time of query compilation like "create table if not exists as select"
        Hide
        Chinna Rao Lalam added a comment -

        Assigned to me as i have patch ready with me. Once the test cases execution completes will upload the same.

        Show
        Chinna Rao Lalam added a comment - Assigned to me as i have patch ready with me. Once the test cases execution completes will upload the same.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2974/
        -----------------------------------------------------------

        Review request for hive and namit jain.

        Summary
        -------

        INSERT OVERWRITE TABLE X PARTITION (a=b, c=d) IF NOT EXISTS ...
        The partition should be created and written if and only if it's not there already.
        Added a check to validate the partition exist or not.

        This addresses bug HIVE-2602.
        https://issues.apache.org/jira/browse/HIVE-2602

        Diffs


        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1206077
        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1206077
        trunk/ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q PRE-CREATION
        trunk/ql/src/test/results/clientpositive/insert1_overwrite_partitions.q.out PRE-CREATION

        Diff: https://reviews.apache.org/r/2974/diff

        Testing
        -------

        all tests passed and added new testcases

        Thanks,

        chinna

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2974/ ----------------------------------------------------------- Review request for hive and namit jain. Summary ------- INSERT OVERWRITE TABLE X PARTITION (a=b, c=d) IF NOT EXISTS ... The partition should be created and written if and only if it's not there already. Added a check to validate the partition exist or not. This addresses bug HIVE-2602 . https://issues.apache.org/jira/browse/HIVE-2602 Diffs trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1206077 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1206077 trunk/ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q PRE-CREATION trunk/ql/src/test/results/clientpositive/insert1_overwrite_partitions.q.out PRE-CREATION Diff: https://reviews.apache.org/r/2974/diff Testing ------- all tests passed and added new testcases Thanks, chinna
        Hide
        Chinna Rao Lalam added a comment -

        Provided a check to validate the partition exist or not if it exists remaining execution it will skip and execution will complete.

        Show
        Chinna Rao Lalam added a comment - Provided a check to validate the partition exist or not if it exists remaining execution it will skip and execution will complete.
        Hide
        Namit Jain added a comment -

        https://reviews.facebook.net/D579

        Thanks Chinna. I will take a look

        Show
        Namit Jain added a comment - https://reviews.facebook.net/D579 Thanks Chinna. I will take a look
        Hide
        Phabricator added a comment -

        njain requested code review of "HIVE-2602 [jira] add support for insert partition overwrite(...) if not exists".
        Reviewers: JIRA

        HIVE-2602

        INSERT OVERWRITE TABLE X PARTITION (a=b, c=d) IF NOT EXISTS ...

        The partition should be created and written if and only if it's not there already.

        The support can be added for dynamic partitions in the future, but this jira is for adding this support for static partitions.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D579

        AFFECTED FILES
        ql/src/test/results/clientpositive/insert1_overwrite_partitions.q.out
        ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q
        ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
        ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/1281/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - njain requested code review of " HIVE-2602 [jira] add support for insert partition overwrite(...) if not exists". Reviewers: JIRA HIVE-2602 INSERT OVERWRITE TABLE X PARTITION (a=b, c=d) IF NOT EXISTS ... The partition should be created and written if and only if it's not there already. The support can be added for dynamic partitions in the future, but this jira is for adding this support for static partitions. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D579 AFFECTED FILES ql/src/test/results/clientpositive/insert1_overwrite_partitions.q.out ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/1281/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Namit Jain added a comment -

        Sorry about the delay on my part for review

        Show
        Namit Jain added a comment - Sorry about the delay on my part for review
        Hide
        Phabricator added a comment -

        njain has commented on the revision "HIVE-2602 [jira] add support for insert partition overwrite(...) if not exists".

        INLINE COMMENTS
        ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:890 fix the spelling of existls
        ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q:1 add a explain plan for the insert overtwrite statements.
        ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:892 Instead of adding a new variable 'skipAnalysis'- add a return value to doPhase1 - continue or get out remaining processing.
        ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q:3 Add a test for insert overwrite if not exists for dynamic partitions
        ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q:6 For a small output, can you load only 5 rows from kv1.txt

        REVISION DETAIL
        https://reviews.facebook.net/D579

        Show
        Phabricator added a comment - njain has commented on the revision " HIVE-2602 [jira] add support for insert partition overwrite(...) if not exists". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:890 fix the spelling of existls ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q:1 add a explain plan for the insert overtwrite statements. ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:892 Instead of adding a new variable 'skipAnalysis'- add a return value to doPhase1 - continue or get out remaining processing. ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q:3 Add a test for insert overwrite if not exists for dynamic partitions ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q:6 For a small output, can you load only 5 rows from kv1.txt REVISION DETAIL https://reviews.facebook.net/D579
        Hide
        Chinna Rao Lalam added a comment -

        Reworked on the patch according to the review comments and patch uploaded in the jira.
        This patch will take care of static partitions only.

        Show
        Chinna Rao Lalam added a comment - Reworked on the patch according to the review comments and patch uploaded in the jira. This patch will take care of static partitions only.
        Hide
        Namit Jain added a comment -

        I am getting a lot of merge conflicts. Can you refresh ?

        Show
        Namit Jain added a comment - I am getting a lot of merge conflicts. Can you refresh ?
        Hide
        Chinna Rao Lalam added a comment -

        Rebased and uploaded the patch

        Show
        Chinna Rao Lalam added a comment - Rebased and uploaded the patch
        Hide
        Phabricator added a comment -

        njain requested code review of "HIVE-2602 [jira] add support for insert partition overwrite(...) if not exists".
        Reviewers: JIRA

        HIVE-2602

        INSERT OVERWRITE TABLE X PARTITION (a=b, c=d) IF NOT EXISTS ...

        The partition should be created and written if and only if it's not there already.

        The support can be added for dynamic partitions in the future, but this jira is for adding this support for static partitions.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D879

        AFFECTED FILES
        ql/src/test/results/clientpositive/insertover_dynapart_ifnotexists.q.out
        ql/src/test/results/clientpositive/insert1_overwrite_partitions.q.out
        ql/src/test/queries/clientpositive/insertover_dynapart_ifnotexists.q
        ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q
        ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
        ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/1863/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - njain requested code review of " HIVE-2602 [jira] add support for insert partition overwrite(...) if not exists". Reviewers: JIRA HIVE-2602 INSERT OVERWRITE TABLE X PARTITION (a=b, c=d) IF NOT EXISTS ... The partition should be created and written if and only if it's not there already. The support can be added for dynamic partitions in the future, but this jira is for adding this support for static partitions. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D879 AFFECTED FILES ql/src/test/results/clientpositive/insertover_dynapart_ifnotexists.q.out ql/src/test/results/clientpositive/insert1_overwrite_partitions.q.out ql/src/test/queries/clientpositive/insertover_dynapart_ifnotexists.q ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/1863/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Namit Jain added a comment -

        1. make the tests deterministic. Add a order by when you are selecting 5 rows (limit 5)
        2. Throw a semantic error for 'if not exists' for dynamic partitions - it might be confusing to document this
        behavior for dynamic partitions, so let us now allow it.

        Show
        Namit Jain added a comment - 1. make the tests deterministic. Add a order by when you are selecting 5 rows (limit 5) 2. Throw a semantic error for 'if not exists' for dynamic partitions - it might be confusing to document this behavior for dynamic partitions, so let us now allow it.
        Hide
        Chinna Rao Lalam added a comment -

        updated the patch as per the review comments.

        Show
        Chinna Rao Lalam added a comment - updated the patch as per the review comments.
        Hide
        Phabricator added a comment -

        njain updated the revision "HIVE-2602 [jira] add support for insert partition overwrite(...) if not exists".
        Reviewers: JIRA

        HIVE-2602

        REVISION DETAIL
        https://reviews.facebook.net/D579

        AFFECTED FILES
        ql/src/test/results/clientnegative/insertover_dynapart_ifnotexists.q.out
        ql/src/test/results/clientpositive/insert1_overwrite_partitions.q.out
        ql/src/test/queries/clientnegative/insertover_dynapart_ifnotexists.q
        ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q
        ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
        ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java
        ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java

        Show
        Phabricator added a comment - njain updated the revision " HIVE-2602 [jira] add support for insert partition overwrite(...) if not exists". Reviewers: JIRA HIVE-2602 REVISION DETAIL https://reviews.facebook.net/D579 AFFECTED FILES ql/src/test/results/clientnegative/insertover_dynapart_ifnotexists.q.out ql/src/test/results/clientpositive/insert1_overwrite_partitions.q.out ql/src/test/queries/clientnegative/insertover_dynapart_ifnotexists.q ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        Hide
        Namit Jain added a comment -

        Committed. Thanks Chinna Rao Lalam

        Show
        Namit Jain added a comment - Committed. Thanks Chinna Rao Lalam
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.23.0 #36 (See https://builds.apache.org/job/Hive-trunk-h0.23.0/36/)
        HIVE-2602 add support for insert partition overwrite(...) if not
        exists (Chinna Rao Lalam via namit)

        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220596
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        • /hive/trunk/ql/src/test/queries/clientnegative/insertover_dynapart_ifnotexists.q
        • /hive/trunk/ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q
        • /hive/trunk/ql/src/test/results/clientnegative/insertover_dynapart_ifnotexists.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/insert1_overwrite_partitions.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.23.0 #36 (See https://builds.apache.org/job/Hive-trunk-h0.23.0/36/ ) HIVE-2602 add support for insert partition overwrite(...) if not exists (Chinna Rao Lalam via namit) namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220596 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/test/queries/clientnegative/insertover_dynapart_ifnotexists.q /hive/trunk/ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q /hive/trunk/ql/src/test/results/clientnegative/insertover_dynapart_ifnotexists.q.out /hive/trunk/ql/src/test/results/clientpositive/insert1_overwrite_partitions.q.out
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1158 (See https://builds.apache.org/job/Hive-trunk-h0.21/1158/)
        HIVE-2602 add support for insert partition overwrite(...) if not
        exists (Chinna Rao Lalam via namit)

        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220596
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        • /hive/trunk/ql/src/test/queries/clientnegative/insertover_dynapart_ifnotexists.q
        • /hive/trunk/ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q
        • /hive/trunk/ql/src/test/results/clientnegative/insertover_dynapart_ifnotexists.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/insert1_overwrite_partitions.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1158 (See https://builds.apache.org/job/Hive-trunk-h0.21/1158/ ) HIVE-2602 add support for insert partition overwrite(...) if not exists (Chinna Rao Lalam via namit) namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220596 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/test/queries/clientnegative/insertover_dynapart_ifnotexists.q /hive/trunk/ql/src/test/queries/clientpositive/insert1_overwrite_partitions.q /hive/trunk/ql/src/test/results/clientnegative/insertover_dynapart_ifnotexists.q.out /hive/trunk/ql/src/test/results/clientpositive/insert1_overwrite_partitions.q.out
        Hide
        Ashutosh Chauhan added a comment -

        This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

        Show
        Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
        Hide
        Phabricator added a comment -

        njain has abandoned the revision "HIVE-2602 [jira] add support for insert partition overwrite(...) if not exists".

        REVISION DETAIL
        https://reviews.facebook.net/D579

        Show
        Phabricator added a comment - njain has abandoned the revision " HIVE-2602 [jira] add support for insert partition overwrite(...) if not exists". REVISION DETAIL https://reviews.facebook.net/D579

          People

          • Assignee:
            Chinna Rao Lalam
            Reporter:
            Namit Jain
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development