Hive
  1. Hive
  2. HIVE-2941

Hive should expand nested structs when setting the table schema from thrift structs

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: None
    • Labels:
      None

      Description

      When setting a table serde, the deserializer is queried for its schema, which is used to set the metastore table schema. The current implementation uses the class name stored in the field as the field type.

      By storing the class name as the field type, users cannot see the contents of a struct with "describe tblname". Applications that query HiveMetaStore for the table schema (specifically HCatalog in this case) see an unknown field type, rather than a struct containing known field types.

      Hive should store the expanded schema in the metastore so users browsing the schema see expanded fields, and applications querying metastore see familiar types.

      DETAILS

      Set the table serde to something like this. This serde uses the built-in ThriftStructObjectInspector.

      alter table foo_test
        set serde "com.twitter.elephantbird.hive.serde.ThriftSerDe"
        with serdeproperties ("serialization.class"="com.foo.Foo");
      

      This causes a call to MetaStoreUtils.getFieldsFromDeserializer which returns a list of fields and their schemas. However, currently it does not handle nested structs, and if com.foo.Foo above contains a field com.foo.Bar, the class name com.foo.Bar would appear as the field type. Instead, nested structs should be expanded.

        Issue Links

          Activity

            People

            • Assignee:
              Travis Crawford
              Reporter:
              Travis Crawford
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development