Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-1068

CREATE VIEW followup: add a "table type" enum attribute in metastore's MTable, and also null out irrelevant attributes for MTable instances which describe views

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.6.0
    • Metastore
    • None
    • Reviewed

    Description

      Zheng's description:

      5. TODO: Metadata change: We store "view" definitions in the same metadata table that we store "table" definitions.
      Shall we add a field "table type" so we know whether it's a table, external table, view, or materialized view in the future.
      We should clean up the additional useless fields in "view" - the test output shows that we are storing some garbage information for views.

      Attachments

        1. HIVE-1068.2.patch
          79 kB
          John Sichi
        2. HIVE-1068.1.patch
          77 kB
          John Sichi

        Issue Links

          Activity

            jvs John Sichi added a comment -

            I found out that the old version of thrift we are using does not support generation of native Java5 enums. Instead, it generates an obsolete type-unsafe enum pattern which will go away when we eventually upgrade thrift.

            So, I'm thinking of just using a String in both thrift and JDO for the table type attribute, and then adding a native Java5 enum to wrap it on levels above thrift.

            This attribute will be null in the persistence for existing metastore instances after upgrade, but I can infer the table type from other attributes (for external tables, we have a EXTERNAL=TRUE property set) and replace the null as part of retrieving the object definition.

            jvs John Sichi added a comment - I found out that the old version of thrift we are using does not support generation of native Java5 enums. Instead, it generates an obsolete type-unsafe enum pattern which will go away when we eventually upgrade thrift. So, I'm thinking of just using a String in both thrift and JDO for the table type attribute, and then adding a native Java5 enum to wrap it on levels above thrift. This attribute will be null in the persistence for existing metastore instances after upgrade, but I can infer the table type from other attributes (for external tables, we have a EXTERNAL=TRUE property set) and replace the null as part of retrieving the object definition.
            jvs John Sichi added a comment -

            Notes for reviewers:

            (1)

            I introduced the new TableType enum in package o.a.h.h.metastore. I didn't put it in metastore.api since it is not a generated file, and I didn't put it in metastore.model since that isn't accessed by most of Hive. If there's a better place to put it, please let me know.

            For the enum values, I used MANAGED_TABLE, EXTERNAL_TABLE, and VIRTUAL_VIEW. This is so that later if we add MATERIALIZED_VIEW, there won't be any confusion. If we eventually use this for implementing the TABLE_TYPE attribute in the ResultSet for JDBC's getTables, we can use a CASE expression to fold these into simpler names (e.g. JDBC uses "TABLE" and "VIEW").

            (2)

            I tried making the SerDe itself null, but that caused problems in too many places that were requiring it to be non-null. So instead I nulled/emptied all of its attributes.

            jvs John Sichi added a comment - Notes for reviewers: (1) I introduced the new TableType enum in package o.a.h.h.metastore. I didn't put it in metastore.api since it is not a generated file, and I didn't put it in metastore.model since that isn't accessed by most of Hive. If there's a better place to put it, please let me know. For the enum values, I used MANAGED_TABLE, EXTERNAL_TABLE, and VIRTUAL_VIEW. This is so that later if we add MATERIALIZED_VIEW, there won't be any confusion. If we eventually use this for implementing the TABLE_TYPE attribute in the ResultSet for JDBC's getTables, we can use a CASE expression to fold these into simpler names (e.g. JDBC uses "TABLE" and "VIEW"). (2) I tried making the SerDe itself null, but that caused problems in too many places that were requiring it to be non-null. So instead I nulled/emptied all of its attributes.
            jvs John Sichi added a comment -

            See the updated output for create_view.q.out for an example of what we'll be storing in the view descriptor after this change.

            jvs John Sichi added a comment - See the updated output for create_view.q.out for an example of what we'll be storing in the view descriptor after this change.
            jvs John Sichi added a comment -

            I have written up metastore upgrade instructions here:

            http://wiki.apache.org/hadoop/Hive/ViewDev#Metastore_Upgrades

            jvs John Sichi added a comment - I have written up metastore upgrade instructions here: http://wiki.apache.org/hadoop/Hive/ViewDev#Metastore_Upgrades
            zshao Zheng Shao added a comment -

            The meaning of isNativeSerDe(String) changed. Can we rename it to "shouldGetColsFromSerDe" (and invert the condition)?

            zshao Zheng Shao added a comment - The meaning of isNativeSerDe(String) changed. Can we rename it to "shouldGetColsFromSerDe" (and invert the condition)?
            jvs John Sichi added a comment -

            .2 patch addresses Zheng's comment and resolves a conflict with trunk.

            jvs John Sichi added a comment - .2 patch addresses Zheng's comment and resolves a conflict with trunk.
            zshao Zheng Shao added a comment -

            Committed. Thanks John!

            zshao Zheng Shao added a comment - Committed. Thanks John!

            People

              jvs John Sichi
              jvs John Sichi
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: