Hive
  1. Hive
  2. HIVE-2825

Concatenating a partition does not inherit location from table

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When a table is created in one dfs, a partition is added to that table, the table's dfs is changed, and the partitioned is concatenated, the partitions location remains exactly the same. Instead, it should inherit its location from the table, and be updated accordingly.

      See https://issues.apache.org/jira/browse/HIVE-1707 for an analogous change to insert overwrite.

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-2825 Concatenating a partition does not inherit location from table
        (Kevin Wilfong via namit) (Revision 1294942)

        Result = ABORTED
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294942
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
        • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyPartitionIsSubdirectoryOfTableHook.java
        • /hive/trunk/ql/src/test/queries/clientpositive/concatenate_inherit_table_location.q
        • /hive/trunk/ql/src/test/results/clientpositive/concatenate_inherit_table_location.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2825 Concatenating a partition does not inherit location from table (Kevin Wilfong via namit) (Revision 1294942) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294942 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyPartitionIsSubdirectoryOfTableHook.java /hive/trunk/ql/src/test/queries/clientpositive/concatenate_inherit_table_location.q /hive/trunk/ql/src/test/results/clientpositive/concatenate_inherit_table_location.q.out
        Hide
        Ashutosh Chauhan added a comment -

        This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

        Show
        Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1284 (See https://builds.apache.org/job/Hive-trunk-h0.21/1284/)
        HIVE-2825 Concatenating a partition does not inherit location from table
        (Kevin Wilfong via namit) (Revision 1294942)

        Result = FAILURE
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294942
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
        • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyPartitionIsSubdirectoryOfTableHook.java
        • /hive/trunk/ql/src/test/queries/clientpositive/concatenate_inherit_table_location.q
        • /hive/trunk/ql/src/test/results/clientpositive/concatenate_inherit_table_location.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1284 (See https://builds.apache.org/job/Hive-trunk-h0.21/1284/ ) HIVE-2825 Concatenating a partition does not inherit location from table (Kevin Wilfong via namit) (Revision 1294942) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294942 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyPartitionIsSubdirectoryOfTableHook.java /hive/trunk/ql/src/test/queries/clientpositive/concatenate_inherit_table_location.q /hive/trunk/ql/src/test/results/clientpositive/concatenate_inherit_table_location.q.out
        Hide
        Namit Jain added a comment -

        Committed. Thanks Kevin.

        Show
        Namit Jain added a comment - Committed. Thanks Kevin.
        Hide
        Phabricator added a comment -

        kevinwilfong has commented on the revision "HIVE-2825 [jira] Concatenating a partition does not inherit location from table".

        I left the stats task in as the number of files should change when concatenating files.

        REVISION DETAIL
        https://reviews.facebook.net/D1995

        Show
        Phabricator added a comment - kevinwilfong has commented on the revision " HIVE-2825 [jira] Concatenating a partition does not inherit location from table". I left the stats task in as the number of files should change when concatenating files. REVISION DETAIL https://reviews.facebook.net/D1995
        Hide
        Phabricator added a comment -

        kevinwilfong updated the revision "HIVE-2825 [jira] Concatenating a partition does not inherit location from table".
        Reviewers: JIRA, njain

        Added a test to verify that when the scheme for the table is changed, concatenating a partition changes the scheme for the partition as well. It uses a hook because the location is masked in the output files.

        REVISION DETAIL
        https://reviews.facebook.net/D1995

        AFFECTED FILES
        ql/src/test/results/clientpositive/concatenate_inherit_table_location.q.out
        ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyPartitionIsSubdirectoryOfTableHook.java
        ql/src/test/queries/clientpositive/concatenate_inherit_table_location.q
        ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java

        Show
        Phabricator added a comment - kevinwilfong updated the revision " HIVE-2825 [jira] Concatenating a partition does not inherit location from table". Reviewers: JIRA, njain Added a test to verify that when the scheme for the table is changed, concatenating a partition changes the scheme for the partition as well. It uses a hook because the location is masked in the output files. REVISION DETAIL https://reviews.facebook.net/D1995 AFFECTED FILES ql/src/test/results/clientpositive/concatenate_inherit_table_location.q.out ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyPartitionIsSubdirectoryOfTableHook.java ql/src/test/queries/clientpositive/concatenate_inherit_table_location.q ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
        Hide
        Phabricator added a comment -

        njain has commented on the revision "HIVE-2825 [jira] Concatenating a partition does not inherit location from table".

        Do we need to collect stats for this ?

        I mean, the number of rows etc. cannot change.

        REVISION DETAIL
        https://reviews.facebook.net/D1995

        Show
        Phabricator added a comment - njain has commented on the revision " HIVE-2825 [jira] Concatenating a partition does not inherit location from table". Do we need to collect stats for this ? I mean, the number of rows etc. cannot change. REVISION DETAIL https://reviews.facebook.net/D1995
        Hide
        Phabricator added a comment -

        njain has commented on the revision "HIVE-2825 [jira] Concatenating a partition does not inherit location from table".

        Can you add a test for this ?
        Check the location of the output via describe.

        REVISION DETAIL
        https://reviews.facebook.net/D1995

        Show
        Phabricator added a comment - njain has commented on the revision " HIVE-2825 [jira] Concatenating a partition does not inherit location from table". Can you add a test for this ? Check the location of the output via describe. REVISION DETAIL https://reviews.facebook.net/D1995
        Hide
        Phabricator added a comment -

        kevinwilfong requested code review of "HIVE-2825 [jira] Concatenating a partition does not inherit location from table".
        Reviewers: JIRA

        https://issues.apache.org/jira/browse/HIVE-2825

        When merging a partition the location is recreated using the scheme and authority of the table's URI, so if the table's default dfs has changed, the partition's location will as well. As part of this I noticed that the auto stats gather configuration variable was being ignored for merge, so I added this as well.

        When a table is created in one dfs, a partition is added to that table, the table's dfs is changed, and the partitioned is concatenated, the partitions location remains exactly the same. Instead, it should inherit its location from the table, and be updated accordingly.

        See https://issues.apache.org/jira/browse/HIVE-1707 for an analogous change to insert overwrite.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D1995

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/4251/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - kevinwilfong requested code review of " HIVE-2825 [jira] Concatenating a partition does not inherit location from table". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2825 When merging a partition the location is recreated using the scheme and authority of the table's URI, so if the table's default dfs has changed, the partition's location will as well. As part of this I noticed that the auto stats gather configuration variable was being ignored for merge, so I added this as well. When a table is created in one dfs, a partition is added to that table, the table's dfs is changed, and the partitioned is concatenated, the partitions location remains exactly the same. Instead, it should inherit its location from the table, and be updated accordingly. See https://issues.apache.org/jira/browse/HIVE-1707 for an analogous change to insert overwrite. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1995 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4251/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          People

          • Assignee:
            Kevin Wilfong
            Reporter:
            Kevin Wilfong
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development