Hive
  1. Hive
  2. HIVE-2795

View partitions do not have a storage descriptor

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: Metastore
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Requires upgrade of the metadata corresponding to partitioned views. Please see the comments in metastore/scripts/upgrade/001-HIVE-2795.update_view_partitions.py for more information.
      Show
      Requires upgrade of the metadata corresponding to partitioned views. Please see the comments in metastore/scripts/upgrade/001- HIVE-2795 .update_view_partitions.py for more information.

      Description

      Besides being an inconsistency, it causes errors.

      Calling describe formatted on a view partition throws an exception

      java.lang.NullPointerException
      at org.apache.hadoop.hive.ql.metadata.Partition.getCols(Partition.java:505)
      at org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:2570)

      because it does not have a column descriptor, which is part of the storage descriptor.

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-2966 :Revert HIVE-2795 (Thejas Nair via Ashutosh Chauhan) (Revision 1328568)
        HIVE-2795 View partitions do not have a storage descriptor
        (Kevin Wilfong via namit) (Revision 1242682)

        Result = ABORTED
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328568
        Files :

        • /hive/trunk/metastore/scripts/upgrade/001-HIVE-2795.update_view_partitions.py
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
        • /hive/trunk/ql/src/test/queries/clientpositive/describe_formatted_view_partitioned_json.q
        • /hive/trunk/ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/describe_formatted_view_partitioned_json.q.out

        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242682
        Files :

        • /hive/trunk/metastore/scripts/upgrade/001-HIVE-2795.update_view_partitions.py
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        • /hive/trunk/ql/src/test/queries/clientpositive/describe_formatted_view_partitioned.q
        • /hive/trunk/ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2966 :Revert HIVE-2795 (Thejas Nair via Ashutosh Chauhan) (Revision 1328568) HIVE-2795 View partitions do not have a storage descriptor (Kevin Wilfong via namit) (Revision 1242682) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328568 Files : /hive/trunk/metastore/scripts/upgrade/001- HIVE-2795 .update_view_partitions.py /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java /hive/trunk/ql/src/test/queries/clientpositive/describe_formatted_view_partitioned_json.q /hive/trunk/ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out /hive/trunk/ql/src/test/results/clientpositive/describe_formatted_view_partitioned_json.q.out namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242682 Files : /hive/trunk/metastore/scripts/upgrade/001- HIVE-2795 .update_view_partitions.py /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java /hive/trunk/ql/src/test/queries/clientpositive/describe_formatted_view_partitioned.q /hive/trunk/ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out
        Hide
        Phabricator added a comment -

        kevinwilfong has closed the revision "HIVE-2795 [jira] View partitions do not have a storage descriptor".

        REVISION DETAIL
        https://reviews.facebook.net/D1683

        To: JIRA, njain, kevinwilfong
        Cc: njain

        Show
        Phabricator added a comment - kevinwilfong has closed the revision " HIVE-2795 [jira] View partitions do not have a storage descriptor". REVISION DETAIL https://reviews.facebook.net/D1683 To: JIRA, njain, kevinwilfong Cc: njain
        Ashutosh Chauhan made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Ashutosh Chauhan added a comment -

        This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

        Show
        Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1387 (See https://builds.apache.org/job/Hive-trunk-h0.21/1387/)
        HIVE-2966 :Revert HIVE-2795 (Thejas Nair via Ashutosh Chauhan) (Revision 1328568)

        Result = SUCCESS
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328568
        Files :

        • /hive/trunk/metastore/scripts/upgrade/001-HIVE-2795.update_view_partitions.py
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
        • /hive/trunk/ql/src/test/queries/clientpositive/describe_formatted_view_partitioned_json.q
        • /hive/trunk/ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/describe_formatted_view_partitioned_json.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1387 (See https://builds.apache.org/job/Hive-trunk-h0.21/1387/ ) HIVE-2966 :Revert HIVE-2795 (Thejas Nair via Ashutosh Chauhan) (Revision 1328568) Result = SUCCESS hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328568 Files : /hive/trunk/metastore/scripts/upgrade/001- HIVE-2795 .update_view_partitions.py /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java /hive/trunk/ql/src/test/queries/clientpositive/describe_formatted_view_partitioned_json.q /hive/trunk/ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out /hive/trunk/ql/src/test/results/clientpositive/describe_formatted_view_partitioned_json.q.out
        Carl Steinbach made changes -
        Release Note Requires upgrade of the metadata corresponding to partitioned views. Please see the comments in metastore/scripts/upgrade/001-HIVE-2795.update_view_partitions.py for more information.
        Carl Steinbach made changes -
        Fix Version/s 0.9.0 [ 12317742 ]
        Component/s Metastore [ 12312584 ]
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1252 (See https://builds.apache.org/job/Hive-trunk-h0.21/1252/)
        HIVE-2795 View partitions do not have a storage descriptor
        (Kevin Wilfong via namit)

        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242682
        Files :

        • /hive/trunk/metastore/scripts/upgrade/001-HIVE-2795.update_view_partitions.py
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        • /hive/trunk/ql/src/test/queries/clientpositive/describe_formatted_view_partitioned.q
        • /hive/trunk/ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1252 (See https://builds.apache.org/job/Hive-trunk-h0.21/1252/ ) HIVE-2795 View partitions do not have a storage descriptor (Kevin Wilfong via namit) namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242682 Files : /hive/trunk/metastore/scripts/upgrade/001- HIVE-2795 .update_view_partitions.py /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java /hive/trunk/ql/src/test/queries/clientpositive/describe_formatted_view_partitioned.q /hive/trunk/ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out
        Namit Jain made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Resolution Fixed [ 1 ]
        Hide
        Namit Jain added a comment -

        Committed. Thanks Kevin

        Show
        Namit Jain added a comment - Committed. Thanks Kevin
        Phabricator made changes -
        Attachment HIVE-2795.D1683.3.patch [ 12514066 ]
        Hide
        Phabricator added a comment -

        kevinwilfong updated the revision "HIVE-2795 [jira] View partitions do not have a storage descriptor".
        Reviewers: JIRA, njain

        Included a Python script to go through the metastore and update all view partitions to have a Storage Descriptor. It does so by dropping the partition and adding the partition. As the only view partition data that can be modified is the name, no data will be lost by doing this. Detailed instructions and warnings are included in the script.

        REVISION DETAIL
        https://reviews.facebook.net/D1683

        AFFECTED FILES
        metastore/scripts/upgrade/001-HIVE-2795.update_view_partitions.py
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out
        ql/src/test/queries/clientpositive/describe_formatted_view_partitioned.q

        Show
        Phabricator added a comment - kevinwilfong updated the revision " HIVE-2795 [jira] View partitions do not have a storage descriptor". Reviewers: JIRA, njain Included a Python script to go through the metastore and update all view partitions to have a Storage Descriptor. It does so by dropping the partition and adding the partition. As the only view partition data that can be modified is the name, no data will be lost by doing this. Detailed instructions and warnings are included in the script. REVISION DETAIL https://reviews.facebook.net/D1683 AFFECTED FILES metastore/scripts/upgrade/001- HIVE-2795 .update_view_partitions.py metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out ql/src/test/queries/clientpositive/describe_formatted_view_partitioned.q
        Hide
        Phabricator added a comment -

        njain has accepted the revision "HIVE-2795 [jira] View partitions do not have a storage descriptor".

        REVISION DETAIL
        https://reviews.facebook.net/D1683

        Show
        Phabricator added a comment - njain has accepted the revision " HIVE-2795 [jira] View partitions do not have a storage descriptor". REVISION DETAIL https://reviews.facebook.net/D1683
        Phabricator made changes -
        Attachment HIVE-2795.D1683.2.patch [ 12514033 ]
        Hide
        Phabricator added a comment -

        kevinwilfong updated the revision "HIVE-2795 [jira] View partitions do not have a storage descriptor".
        Reviewers: JIRA, njain

        Added a test to verify calling describe formatted on a view partition succeeds.

        REVISION DETAIL
        https://reviews.facebook.net/D1683

        AFFECTED FILES
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out
        ql/src/test/queries/clientpositive/describe_formatted_view_partitioned.q

        Show
        Phabricator added a comment - kevinwilfong updated the revision " HIVE-2795 [jira] View partitions do not have a storage descriptor". Reviewers: JIRA, njain Added a test to verify calling describe formatted on a view partition succeeds. REVISION DETAIL https://reviews.facebook.net/D1683 AFFECTED FILES metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out ql/src/test/queries/clientpositive/describe_formatted_view_partitioned.q
        Kevin Wilfong made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Phabricator made changes -
        Attachment HIVE-2795.D1683.1.patch [ 12514030 ]
        Hide
        Phabricator added a comment -

        kevinwilfong requested code review of "HIVE-2795 [jira] View partitions do not have a storage descriptor".
        Reviewers: JIRA

        https://issues.apache.org/jira/browse/HIVE-2795

        Updated the metastore so that if a partition is added to a view and the partition's StorageDescriptor is null it will use a copy of the table's StorageDescriptor.

        Besides being an inconsistency, it causes errors.

        Calling describe formatted on a view partition throws an exception

        java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.metadata.Partition.getCols(Partition.java:505)
        at org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:2570)

        because it does not have a column descriptor, which is part of the storage descriptor.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D1683

        AFFECTED FILES
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/3591/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - kevinwilfong requested code review of " HIVE-2795 [jira] View partitions do not have a storage descriptor". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2795 Updated the metastore so that if a partition is added to a view and the partition's StorageDescriptor is null it will use a copy of the table's StorageDescriptor. Besides being an inconsistency, it causes errors. Calling describe formatted on a view partition throws an exception java.lang.NullPointerException at org.apache.hadoop.hive.ql.metadata.Partition.getCols(Partition.java:505) at org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:2570) because it does not have a column descriptor, which is part of the storage descriptor. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1683 AFFECTED FILES metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/3591/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Kevin Wilfong made changes -
        Field Original Value New Value
        Description Besides being an inconsistency, it causes errors.

        Calling describe formatted on a view partition throws an exception

        java.lang.NullPointerException
          at org.apache.hadoop.hive.ql.metadata.Partition.getCols(Partition.java:505)
          at org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:2570)

        because it does not have a column descriptor, which is part of the storage descriptor.
        Namit Jain created issue -

          People

          • Assignee:
            Kevin Wilfong
            Reporter:
            Namit Jain
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development