Hive
  1. Hive
  2. HIVE-1068

CREATE VIEW followup: add a "table type" enum attribute in metastore's MTable, and also null out irrelevant attributes for MTable instances which describe views

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Component/s: Metastore
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Zheng's description:

      5. TODO: Metadata change: We store "view" definitions in the same metadata table that we store "table" definitions.
      Shall we add a field "table type" so we know whether it's a table, external table, view, or materialized view in the future.
      We should clean up the additional useless fields in "view" - the test output shows that we are storing some garbage information for views.

      1. HIVE-1068.1.patch
        77 kB
        John Sichi
      2. HIVE-1068.2.patch
        79 kB
        John Sichi

        Issue Links

          Activity

          Hide
          Zheng Shao added a comment -

          Committed. Thanks John!

          Show
          Zheng Shao added a comment - Committed. Thanks John!
          Hide
          John Sichi added a comment -

          .2 patch addresses Zheng's comment and resolves a conflict with trunk.

          Show
          John Sichi added a comment - .2 patch addresses Zheng's comment and resolves a conflict with trunk.
          Hide
          Zheng Shao added a comment -

          The meaning of isNativeSerDe(String) changed. Can we rename it to "shouldGetColsFromSerDe" (and invert the condition)?

          Show
          Zheng Shao added a comment - The meaning of isNativeSerDe(String) changed. Can we rename it to "shouldGetColsFromSerDe" (and invert the condition)?
          Hide
          John Sichi added a comment -

          I have written up metastore upgrade instructions here:

          http://wiki.apache.org/hadoop/Hive/ViewDev#Metastore_Upgrades

          Show
          John Sichi added a comment - I have written up metastore upgrade instructions here: http://wiki.apache.org/hadoop/Hive/ViewDev#Metastore_Upgrades
          Hide
          John Sichi added a comment -

          See the updated output for create_view.q.out for an example of what we'll be storing in the view descriptor after this change.

          Show
          John Sichi added a comment - See the updated output for create_view.q.out for an example of what we'll be storing in the view descriptor after this change.
          Hide
          John Sichi added a comment -

          Notes for reviewers:

          (1)

          I introduced the new TableType enum in package o.a.h.h.metastore. I didn't put it in metastore.api since it is not a generated file, and I didn't put it in metastore.model since that isn't accessed by most of Hive. If there's a better place to put it, please let me know.

          For the enum values, I used MANAGED_TABLE, EXTERNAL_TABLE, and VIRTUAL_VIEW. This is so that later if we add MATERIALIZED_VIEW, there won't be any confusion. If we eventually use this for implementing the TABLE_TYPE attribute in the ResultSet for JDBC's getTables, we can use a CASE expression to fold these into simpler names (e.g. JDBC uses "TABLE" and "VIEW").

          (2)

          I tried making the SerDe itself null, but that caused problems in too many places that were requiring it to be non-null. So instead I nulled/emptied all of its attributes.

          Show
          John Sichi added a comment - Notes for reviewers: (1) I introduced the new TableType enum in package o.a.h.h.metastore. I didn't put it in metastore.api since it is not a generated file, and I didn't put it in metastore.model since that isn't accessed by most of Hive. If there's a better place to put it, please let me know. For the enum values, I used MANAGED_TABLE, EXTERNAL_TABLE, and VIRTUAL_VIEW. This is so that later if we add MATERIALIZED_VIEW, there won't be any confusion. If we eventually use this for implementing the TABLE_TYPE attribute in the ResultSet for JDBC's getTables, we can use a CASE expression to fold these into simpler names (e.g. JDBC uses "TABLE" and "VIEW"). (2) I tried making the SerDe itself null, but that caused problems in too many places that were requiring it to be non-null. So instead I nulled/emptied all of its attributes.
          Hide
          John Sichi added a comment -

          I found out that the old version of thrift we are using does not support generation of native Java5 enums. Instead, it generates an obsolete type-unsafe enum pattern which will go away when we eventually upgrade thrift.

          So, I'm thinking of just using a String in both thrift and JDO for the table type attribute, and then adding a native Java5 enum to wrap it on levels above thrift.

          This attribute will be null in the persistence for existing metastore instances after upgrade, but I can infer the table type from other attributes (for external tables, we have a EXTERNAL=TRUE property set) and replace the null as part of retrieving the object definition.

          Show
          John Sichi added a comment - I found out that the old version of thrift we are using does not support generation of native Java5 enums. Instead, it generates an obsolete type-unsafe enum pattern which will go away when we eventually upgrade thrift. So, I'm thinking of just using a String in both thrift and JDO for the table type attribute, and then adding a native Java5 enum to wrap it on levels above thrift. This attribute will be null in the persistence for existing metastore instances after upgrade, but I can infer the table type from other attributes (for external tables, we have a EXTERNAL=TRUE property set) and replace the null as part of retrieving the object definition.

            People

            • Assignee:
              John Sichi
              Reporter:
              John Sichi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development