Hive
  1. Hive
  2. HIVE-2961

Remove need for storage descriptors for view partitions

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None

      Description

      Storage descriptors were introduced for view partitions as part of HIVE-2795. This was to allow view partitions to have the concept of a region as well as to fix a NPE that resulted from calling describe formatted on them.

      Since regions are no longer necessary for view partitions and the NPE can be fixed by not displaying storage information for view partitions (or displaying the view's storage information if this is preferred, although, since a view partition is purely metadata, this does not seem necessary), these are no longer needed.

      This also means the Python script added which retroactively adds storage descriptors to existing view partitions can be removed.

        Activity

        Hide
        Ashutosh Chauhan added a comment -

        This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

        Show
        Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
        Hide
        Ashutosh Chauhan added a comment -

        Fixed via HIVE-2966

        Show
        Ashutosh Chauhan added a comment - Fixed via HIVE-2966
        Hide
        Carl Steinbach added a comment -

        Do you mean delete the content of the file, thus effectively leaving empty files, since there is no upgrade needed when we revert those patches?

        Yup, that's basically what I meant, but now that I think more about it I agree that it doesn't make much sense.

        Show
        Carl Steinbach added a comment - Do you mean delete the content of the file, thus effectively leaving empty files, since there is no upgrade needed when we revert those patches? Yup, that's basically what I meant, but now that I think more about it I agree that it doesn't make much sense.
        Hide
        Ashutosh Chauhan added a comment -

        Canceling patch since current patch may not be valid after HIVE-2795 is reverted.

        Show
        Ashutosh Chauhan added a comment - Canceling patch since current patch may not be valid after HIVE-2795 is reverted.
        Hide
        Ashutosh Chauhan added a comment -

        But if we end up taking that route, I think we should leave the upgrade-0.8.0-to-0.9.0.xxx.sql scripts for the sake of consistency.

        Do you mean delete the content of the file, thus effectively leaving empty files, since there is no upgrade needed when we revert those patches?

        Show
        Ashutosh Chauhan added a comment - But if we end up taking that route, I think we should leave the upgrade-0.8.0-to-0.9.0.xxx.sql scripts for the sake of consistency. Do you mean delete the content of the file, thus effectively leaving empty files, since there is no upgrade needed when we revert those patches?
        Hide
        Kevin Wilfong added a comment -

        We can revert those two patches instead. This patch does fix a null pointer exception that occurs for DESCRIBE FORMATTED on a view partition. But I can submit that in a separate patch.

        Show
        Kevin Wilfong added a comment - We can revert those two patches instead. This patch does fix a null pointer exception that occurs for DESCRIBE FORMATTED on a view partition. But I can submit that in a separate patch.
        Hide
        Kevin Wilfong added a comment -

        We can revert those two patches instead. This patch does fix a null pointer exception that occurs for DESCRIBE FORMATTED on a view partition. But I can submit that in a separate patch.

        Show
        Kevin Wilfong added a comment - We can revert those two patches instead. This patch does fix a null pointer exception that occurs for DESCRIBE FORMATTED on a view partition. But I can submit that in a separate patch.
        Hide
        Carl Steinbach added a comment -

        But if we end up taking that route, I think we should leave the upgrade-0.8.0-to-0.9.0.xxx.sql scripts for the sake of consistency.

        Show
        Carl Steinbach added a comment - But if we end up taking that route, I think we should leave the upgrade-0.8.0-to-0.9.0.xxx.sql scripts for the sake of consistency.
        Hide
        Carl Steinbach added a comment -

        Based on the discussion at today's contrib meeting it sounds like we can drop this patch and instead backout HIVE-2795 and HIVE-2612. Does that sound good?

        Show
        Carl Steinbach added a comment - Based on the discussion at today's contrib meeting it sounds like we can drop this patch and instead backout HIVE-2795 and HIVE-2612 . Does that sound good?
        Hide
        Phabricator added a comment -

        ashutoshc has accepted the revision "HIVE-2961 [jira] Remove need for storage descriptors for view partitions".

        +1 will commit if tests pass.

        REVISION DETAIL
        https://reviews.facebook.net/D2877

        BRANCH
        svn

        Show
        Phabricator added a comment - ashutoshc has accepted the revision " HIVE-2961 [jira] Remove need for storage descriptors for view partitions". +1 will commit if tests pass. REVISION DETAIL https://reviews.facebook.net/D2877 BRANCH svn
        Hide
        Ashutosh Chauhan added a comment -

        I agree with Kevin. Since, views are purely metadata it doesnt make much sense to have a storage-descriptor associated with them.

        Show
        Ashutosh Chauhan added a comment - I agree with Kevin. Since, views are purely metadata it doesnt make much sense to have a storage-descriptor associated with them.
        Hide
        Phabricator added a comment -

        kevinwilfong requested code review of "HIVE-2961 [jira] Remove need for storage descriptors for view partitions".
        Reviewers: JIRA

        https://issues.apache.org/jira/browse/HIVE-2961

        Removed the need for storage descriptors for view partitions and the script to add them.

        Storage descriptors were introduced for view partitions as part of HIVE-2795. This was to allow view partitions to have the concept of a region as well as to fix a NPE that resulted from calling describe formatted on them.

        Since regions are no longer necessary for view partitions and the NPE can be fixed by not displaying storage information for view partitions (or displaying the view's storage information if this is preferred, although, since a view partition is purely metadata, this does not seem necessary), these are no longer needed.

        This also means the Python script added which retroactively adds storage descriptors to existing view partitions can be removed.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D2877

        AFFECTED FILES
        metastore/scripts/upgrade/001-HIVE-2795.update_view_partitions.py
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
        ql/src/test/results/clientpositive/describe_formatted_view_partitioned_json.q.out
        ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out
        ql/src/test/queries/clientpositive/describe_formatted_view_partitioned_json.q
        ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
        ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/6525/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - kevinwilfong requested code review of " HIVE-2961 [jira] Remove need for storage descriptors for view partitions". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2961 Removed the need for storage descriptors for view partitions and the script to add them. Storage descriptors were introduced for view partitions as part of HIVE-2795 . This was to allow view partitions to have the concept of a region as well as to fix a NPE that resulted from calling describe formatted on them. Since regions are no longer necessary for view partitions and the NPE can be fixed by not displaying storage information for view partitions (or displaying the view's storage information if this is preferred, although, since a view partition is purely metadata, this does not seem necessary), these are no longer needed. This also means the Python script added which retroactively adds storage descriptors to existing view partitions can be removed. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2877 AFFECTED FILES metastore/scripts/upgrade/001- HIVE-2795 .update_view_partitions.py metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java ql/src/test/results/clientpositive/describe_formatted_view_partitioned_json.q.out ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out ql/src/test/queries/clientpositive/describe_formatted_view_partitioned_json.q ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/6525/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          People

          • Assignee:
            Kevin Wilfong
            Reporter:
            Kevin Wilfong
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development