Hive
  1. Hive
  2. HIVE-2950

Hive should store the full table schema in partition storage descriptors

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Not A Problem
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Hive tables have a schema, which is copied into the partition storage descriptor when adding a partition. Currently only columns stored in the table storage descriptor are copied - columns that are reported by the serde are not copied. Instead of copying the table storage descriptor columns into the partition columns, the full table schema should be copied.

      DETAILS

      This is a little long but is necessary to show 3 things: current behavior when explicitly listing columns, behavior with HIVE-2941 patched in and serde reported columns, and finally the behavior with this patch (full table schema copied into the partition storage descriptor).

      Here's an example of what currently happens. Note the following:

      • the two manually-defined fields defined for the table are listed in the table storage descriptor.
      • both fields are present in the partition storage descriptor

      This works great because users who query for a partition can look at its storage descriptor and get the schema.

      hive> create external table foo_test (name string, age int) partitioned by (part_dt string);
      hive> describe extended foo_test;
      OK
      name	string	
      age	int	
      part_dt	string	
      	 	 
      Detailed Table Information	Table(tableName:foo_test, dbName:travis_test, owner:travis, createTime:1334256062, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:name, type:string, comment:null), FieldSchema(name:age, type:int, comment:null), FieldSchema(name:part_dt, type:string, comment:null)], location:hdfs://foo.com/warehouse/travis_test.db/foo_test, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), partitionKeys:[FieldSchema(name:part_dt, type:string, comment:null)], parameters:{EXTERNAL=TRUE, transient_lastDdlTime=1334256062}, viewOriginalText:null, viewExpandedText:null, tableType:EXTERNAL_TABLE)	
      Time taken: 0.082 seconds
      
      hive> alter table foo_test add partition (part_dt = '20120331T000000Z') location 'hdfs://foo.com/foo/2012/03/31/00';
      hive> describe extended foo_test partition (part_dt = '20120331T000000Z');
      OK
      name	string	
      age	int	
      part_dt	string	
      	 	 
      Detailed Partition Information	Partition(values:[20120331T000000Z], dbName:travis_test, tableName:foo_test, createTime:1334256131, lastAccessTime:0, sd:StorageDescriptor(cols:[FieldSchema(name:name, type:string, comment:null), FieldSchema(name:age, type:int, comment:null), FieldSchema(name:part_dt, type:string, comment:null)], location:hdfs://foo.com/foo/2012/03/31/00, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), parameters:{transient_lastDdlTime=1334256131})	
      

      CURRENT BEHAVIOR WITH HIVE-2941 PATCHED IN

      Now let's examine what happens when creating a table when the serde reports the schema. Notice the following:

      • The table storage descriptor contains an empty list of columns. However, the table schema is available from the serde reflecting on the serialization class.
      • The partition storage descriptor does contain a single "part_dt" column that was copied from the table partition keys. The actual data columns are not present.
      hive> create external table travis_test.person_test partitioned by (part_dt string) row format serde "com.twitter.elephantbird.hive.serde.ThriftSerDe" with serdeproperties ("serialization.class"="com.twitter.elephantbird.examples.thrift.Person") stored as inputformat "com.twitter.elephantbird.mapred.input.HiveMultiInputFormat" outputformat "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat";
      OK
      Time taken: 0.08 seconds
      hive> describe extended person_test;
      OK
      name	struct<first_name:string,last_name:string>	from deserializer
      id	int	from deserializer
      email	string	from deserializer
      phones	array<struct<number:string,type:struct<value:int>>>	from deserializer
      part_dt	string	
      	 	 
      Detailed Table Information	Table(tableName:person_test, dbName:travis_test, owner:travis, createTime:1334256942, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[], location:hdfs://foo.com/warehouse/travis_test.db/person_test, inputFormat:com.twitter.elephantbird.mapred.input.HiveMultiInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:com.twitter.elephantbird.hive.serde.ThriftSerDe, parameters:{serialization.class=com.twitter.elephantbird.examples.thrift.Person, serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), partitionKeys:[FieldSchema(name:part_dt, type:string, comment:null)], parameters:{EXTERNAL=TRUE, transient_lastDdlTime=1334256942}, viewOriginalText:null, viewExpandedText:null, tableType:EXTERNAL_TABLE)	
      Time taken: 0.147 seconds
      hive> alter table person_test add partition (part_dt = '20120331T000000Z') location 'hdfs://foo.com/foo/2012/03/31/00'; 
      OK
      Time taken: 0.149 seconds
      hive> describe extended person_test partition (part_dt = '20120331T000000Z');
      OK
      part_dt	string	
      	 	 
      Detailed Partition Information	Partition(values:[20120331T000000Z], dbName:travis_test, tableName:person_test, createTime:1334257029, lastAccessTime:0, sd:StorageDescriptor(cols:[FieldSchema(name:part_dt, type:string, comment:null)], location:hdfs://foo.com/foo/2012/03/31/00, inputFormat:com.twitter.elephantbird.mapred.input.HiveMultiInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:com.twitter.elephantbird.hive.serde.ThriftSerDe, parameters:{serialization.class=com.twitter.elephantbird.examples.thrift.Person, serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), parameters:{transient_lastDdlTime=1334257029})	
      Time taken: 0.106 seconds
      hive> 
      

      PROPOSED BEHAVIOR

      I believe the correct thing to do is copy the full table schema (serde-reported columns + partition keys) into the partition storage descriptor. Notice the following:

      • Table storage descriptor does not contain any columns, because they are reported by the serde.
      • Partition storage descriptor now contains both the serde-reported schema, and full table schema.
      hive> create external table travis_test.person_test partitioned by (part_dt string) row format serde "com.twitter.elephantbird.hive.serde.ThriftSerDe" with serdeproperties ("serialization.class"="com.twitter.elephantbird.examples.thrift.Person") stored as inputformat "com.twitter.elephantbird.mapred.input.HiveMultiInputFormat" outputformat "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat";
      OK
      Time taken: 0.076 seconds
      hive> describe extended person_test;                                                                                                                                     OK                                                                                                                                                                       name    struct<first_name:string,last_name:string>      from deserializer
      id	int	from deserializer
      email	string	from deserializer
      phones	array<struct<number:string,type:struct<value:int>>>	from deserializer
      part_dt	string	
      	 	 
      Detailed Table Information	Table(tableName:person_test, dbName:travis_test, owner:travis, createTime:1334257489, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[], location:hdfs://foo.com/warehouse/travis_test.db/person_test, inputFormat:com.twitter.elephantbird.mapred.input.HiveMultiInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:com.twitter.elephantbird.hive.serde.ThriftSerDe, parameters:{serialization.class=com.twitter.elephantbird.examples.thrift.Person, serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), partitionKeys:[FieldSchema(name:part_dt, type:string, comment:null)], parameters:{EXTERNAL=TRUE, transient_lastDdlTime=1334257489}, viewOriginalText:null, viewExpandedText:null, tableType:EXTERNAL_TABLE)	
      Time taken: 0.155 seconds
      hive> alter table person_test add partition (part_dt = '20120331T000000Z') location 'hdfs://foo.com/foo/2012/03/31/00';
      OK                                                                                                                                                                       Time taken: 0.296 seconds                                        
      hive> describe extended person_test partition (part_dt = '20120331T000000Z');                                                                                            OK                                                                                                                                                                       name    struct<first_name:string,last_name:string>      from deserializer
      id	int	from deserializer
      email	string	from deserializer
      phones	array<struct<number:string,type:struct<value:int>>>	from deserializer
      part_dt	string	
      	 	 
      Detailed Partition Information	Partition(values:[20120331T000000Z], dbName:travis_test, tableName:person_test, createTime:1334257504, lastAccessTime:0, sd:StorageDescriptor(cols:[FieldSchema(name:name, type:struct<first_name:string,last_name:string>, comment:from deserializer), FieldSchema(name:id, type:int, comment:from deserializer), FieldSchema(name:email, type:string, comment:from deserializer), FieldSchema(name:phones, type:array<struct<number:string,type:struct<value:int>>>, comment:from deserializer), FieldSchema(name:part_dt, type:string, comment:null)], location:hdfs://foo.com/foo/2012/03/31/00, inputFormat:com.twitter.elephantbird.mapred.input.HiveMultiInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:com.twitter.elephantbird.hive.serde.ThriftSerDe, parameters:{serialization.class=com.twitter.elephantbird.examples.thrift.Person, serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), parameters:{transient_lastDdlTime=1334257504})	
      Time taken: 0.133 seconds
      hive> 
      

        Issue Links

          Activity

          Hide
          Phabricator added a comment -

          travis requested code review of "HIVE-2950 [jira] Hive should store the full table schema in partition storage descriptors".
          Reviewers: JIRA

          When adding a partition, copy the full table schema into the partition storage descriptor. The table storage descriptor may not contain the full schema when using a serde that reports its schema. This change makes partitions with serde-reported schema behave the same as partitions without a serde-reported schem.

          Hive tables have a schema, which is copied into the partition storage descriptor when adding a partition. Currently only columns stored in the table storage descriptor are copied - columns that are reported by the serde are not copied. Instead of copying the table storage descriptor columns into the partition columns, the full table schema should be copied.

          DETAILS

          This is a little long but is necessary to show 3 things: current behavior when explicitly listing columns, behavior with HIVE-2941 patched in and serde reported columns, and finally the behavior with this patch (full table schema copied into the partition storage descriptor).

          Here's an example of what currently happens. Note the following:

          the two manually-defined fields defined for the table are listed in the table storage descriptor.
          both fields are present in the partition storage descriptor

          This works great because users who query for a partition can look at its storage descriptor and get the schema.

          hive> create external table foo_test (name string, age int) partitioned by (part_dt string);
          hive> describe extended foo_test;
          OK
          name string
          age int
          part_dt string

          Detailed Table Information Table(tableName:foo_test, dbName:travis_test, owner:travis, createTime:1334256062, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:name, type:string, comment:null), FieldSchema(name:age, type:int, comment:null), FieldSchema(name:part_dt, type:string, comment:null)], location:hdfs://foo.com/warehouse/travis_test.db/foo_test, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:

          {serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), partitionKeys:[FieldSchema(name:part_dt, type:string, comment:null)], parameters:{EXTERNAL=TRUE, transient_lastDdlTime=1334256062}, viewOriginalText:null, viewExpandedText:null, tableType:EXTERNAL_TABLE)
          Time taken: 0.082 seconds

          hive> alter table foo_test add partition (part_dt = '20120331T000000Z') location 'hdfs://foo.com/foo/2012/03/31/00';
          hive> describe extended foo_test partition (part_dt = '20120331T000000Z');
          OK
          name string
          age int
          part_dt string

          Detailed Partition Information Partition(values:[20120331T000000Z], dbName:travis_test, tableName:foo_test, createTime:1334256131, lastAccessTime:0, sd:StorageDescriptor(cols:[FieldSchema(name:name, type:string, comment:null), FieldSchema(name:age, type:int, comment:null), FieldSchema(name:part_dt, type:string, comment:null)], location:hdfs://foo.com/foo/2012/03/31/00, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}

          ), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), parameters:

          {transient_lastDdlTime=1334256131}

          )

          CURRENT BEHAVIOR WITH HIVE-2941 PATCHED IN

          Now let's examine what happens when creating a table when the serde reports the schema. Notice the following:

          The table storage descriptor contains an empty list of columns. However, the table schema is available from the serde reflecting on the serialization class.
          The partition storage descriptor does contain a single "part_dt" column that was copied from the table partition keys. The actual data columns are not present.

          hive> create external table travis_test.person_test partitioned by (part_dt string) row format serde "com.twitter.elephantbird.hive.serde.ThriftSerDe" with serdeproperties ("serialization.class"="com.twitter.elephantbird.examples.thrift.Person") stored as inputformat "com.twitter.elephantbird.mapred.input.HiveMultiInputFormat" outputformat "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat";
          OK
          Time taken: 0.08 seconds
          hive> describe extended person_test;
          OK
          name struct<first_name:string,last_name:string> from deserializer
          id int from deserializer
          email string from deserializer
          phones array<struct<number:string,type:struct<value:int>>> from deserializer
          part_dt string

          Detailed Table Information Table(tableName:person_test, dbName:travis_test, owner:travis, createTime:1334256942, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[], location:hdfs://foo.com/warehouse/travis_test.db/person_test, inputFormat:com.twitter.elephantbird.mapred.input.HiveMultiInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:com.twitter.elephantbird.hive.serde.ThriftSerDe, parameters:

          {serialization.class=com.twitter.elephantbird.examples.thrift.Person, serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), partitionKeys:[FieldSchema(name:part_dt, type:string, comment:null)], parameters:{EXTERNAL=TRUE, transient_lastDdlTime=1334256942}, viewOriginalText:null, viewExpandedText:null, tableType:EXTERNAL_TABLE)
          Time taken: 0.147 seconds
          hive> alter table person_test add partition (part_dt = '20120331T000000Z') location 'hdfs://foo.com/foo/2012/03/31/00';
          OK
          Time taken: 0.149 seconds
          hive> describe extended person_test partition (part_dt = '20120331T000000Z');
          OK
          part_dt string

          Detailed Partition Information Partition(values:[20120331T000000Z], dbName:travis_test, tableName:person_test, createTime:1334257029, lastAccessTime:0, sd:StorageDescriptor(cols:[FieldSchema(name:part_dt, type:string, comment:null)], location:hdfs://foo.com/foo/2012/03/31/00, inputFormat:com.twitter.elephantbird.mapred.input.HiveMultiInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:com.twitter.elephantbird.hive.serde.ThriftSerDe, parameters:{serialization.class=com.twitter.elephantbird.examples.thrift.Person, serialization.format=1}

          ), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), parameters:

          {transient_lastDdlTime=1334257029}

          )
          Time taken: 0.106 seconds
          hive>

          PROPOSED BEHAVIOR

          I believe the correct thing to do is copy the full table schema (serde-reported columns + partition keys) into the partition storage descriptor. Notice the following:

          Table storage descriptor does not contain any columns, because they are reported by the serde.
          Partition storage descriptor now contains both the serde-reported schema, and full table schema.

          hive> create external table travis_test.person_test partitioned by (part_dt string) row format serde "com.twitter.elephantbird.hive.serde.ThriftSerDe" with serdeproperties ("serialization.class"="com.twitter.elephantbird.examples.thrift.Person") stored as inputformat "com.twitter.elephantbird.mapred.input.HiveMultiInputFormat" outputformat "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat";
          OK
          Time taken: 0.076 seconds
          hive> describe extended person_test; OK name struct<first_name:string,last_name:string> from deserializer
          id int from deserializer
          email string from deserializer
          phones array<struct<number:string,type:struct<value:int>>> from deserializer
          part_dt string

          Detailed Table Information Table(tableName:person_test, dbName:travis_test, owner:travis, createTime:1334257489, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[], location:hdfs://foo.com/warehouse/travis_test.db/person_test, inputFormat:com.twitter.elephantbird.mapred.input.HiveMultiInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:com.twitter.elephantbird.hive.serde.ThriftSerDe, parameters:

          {serialization.class=com.twitter.elephantbird.examples.thrift.Person, serialization.format=1}

          ), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), partitionKeys:[FieldSchema(name:part_dt, type:string, comment:null)], parameters:

          {EXTERNAL=TRUE, transient_lastDdlTime=1334257489}

          , viewOriginalText:null, viewExpandedText:null, tableType:EXTERNAL_TABLE)
          Time taken: 0.155 seconds
          hive> alter table person_test add partition (part_dt = '20120331T000000Z') location 'hdfs://foo.com/foo/2012/03/31/00';
          OK Time taken: 0.296 seconds
          hive> describe extended person_test partition (part_dt = '20120331T000000Z'); OK name struct<first_name:string,last_name:string> from deserializer
          id int from deserializer
          email string from deserializer
          phones array<struct<number:string,type:struct<value:int>>> from deserializer
          part_dt string

          Detailed Partition Information Partition(values:[20120331T000000Z], dbName:travis_test, tableName:person_test, createTime:1334257504, lastAccessTime:0, sd:StorageDescriptor(cols:[FieldSchema(name:name, type:struct<first_name:string,last_name:string>, comment:from deserializer), FieldSchema(name:id, type:int, comment:from deserializer), FieldSchema(name:email, type:string, comment:from deserializer), FieldSchema(name:phones, type:array<struct<number:string,type:struct<value:int>>>, comment:from deserializer), FieldSchema(name:part_dt, type:string, comment:null)], location:hdfs://foo.com/foo/2012/03/31/00, inputFormat:com.twitter.elephantbird.mapred.input.HiveMultiInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:com.twitter.elephantbird.hive.serde.ThriftSerDe, parameters:

          {serialization.class=com.twitter.elephantbird.examples.thrift.Person, serialization.format =1}

          ), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), parameters:

          {transient_lastDdlTime=1334257504}

          )
          Time taken: 0.133 seconds
          hive>

          TEST PLAN
          empty

          REVISION DETAIL
          https://reviews.facebook.net/D2769

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/6297/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - travis requested code review of " HIVE-2950 [jira] Hive should store the full table schema in partition storage descriptors". Reviewers: JIRA When adding a partition, copy the full table schema into the partition storage descriptor. The table storage descriptor may not contain the full schema when using a serde that reports its schema. This change makes partitions with serde-reported schema behave the same as partitions without a serde-reported schem. Hive tables have a schema, which is copied into the partition storage descriptor when adding a partition. Currently only columns stored in the table storage descriptor are copied - columns that are reported by the serde are not copied. Instead of copying the table storage descriptor columns into the partition columns, the full table schema should be copied. DETAILS This is a little long but is necessary to show 3 things: current behavior when explicitly listing columns, behavior with HIVE-2941 patched in and serde reported columns, and finally the behavior with this patch (full table schema copied into the partition storage descriptor). Here's an example of what currently happens. Note the following: the two manually-defined fields defined for the table are listed in the table storage descriptor. both fields are present in the partition storage descriptor This works great because users who query for a partition can look at its storage descriptor and get the schema. hive> create external table foo_test (name string, age int) partitioned by (part_dt string); hive> describe extended foo_test; OK name string age int part_dt string Detailed Table Information Table(tableName:foo_test, dbName:travis_test, owner:travis, createTime:1334256062, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols: [FieldSchema(name:name, type:string, comment:null), FieldSchema(name:age, type:int, comment:null), FieldSchema(name:part_dt, type:string, comment:null)] , location:hdfs://foo.com/warehouse/travis_test.db/foo_test, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters: {serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), partitionKeys: [FieldSchema(name:part_dt, type:string, comment:null)] , parameters:{EXTERNAL=TRUE, transient_lastDdlTime=1334256062}, viewOriginalText:null, viewExpandedText:null, tableType:EXTERNAL_TABLE) Time taken: 0.082 seconds hive> alter table foo_test add partition (part_dt = '20120331T000000Z') location 'hdfs://foo.com/foo/2012/03/31/00'; hive> describe extended foo_test partition (part_dt = '20120331T000000Z'); OK name string age int part_dt string Detailed Partition Information Partition(values: [20120331T000000Z] , dbName:travis_test, tableName:foo_test, createTime:1334256131, lastAccessTime:0, sd:StorageDescriptor(cols: [FieldSchema(name:name, type:string, comment:null), FieldSchema(name:age, type:int, comment:null), FieldSchema(name:part_dt, type:string, comment:null)] , location:hdfs://foo.com/foo/2012/03/31/00, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1} ), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), parameters: {transient_lastDdlTime=1334256131} ) CURRENT BEHAVIOR WITH HIVE-2941 PATCHED IN Now let's examine what happens when creating a table when the serde reports the schema. Notice the following: The table storage descriptor contains an empty list of columns. However, the table schema is available from the serde reflecting on the serialization class. The partition storage descriptor does contain a single "part_dt" column that was copied from the table partition keys. The actual data columns are not present. hive> create external table travis_test.person_test partitioned by (part_dt string) row format serde "com.twitter.elephantbird.hive.serde.ThriftSerDe" with serdeproperties ("serialization.class"="com.twitter.elephantbird.examples.thrift.Person") stored as inputformat "com.twitter.elephantbird.mapred.input.HiveMultiInputFormat" outputformat "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"; OK Time taken: 0.08 seconds hive> describe extended person_test; OK name struct<first_name:string,last_name:string> from deserializer id int from deserializer email string from deserializer phones array<struct<number:string,type:struct<value:int>>> from deserializer part_dt string Detailed Table Information Table(tableName:person_test, dbName:travis_test, owner:travis, createTime:1334256942, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[], location:hdfs://foo.com/warehouse/travis_test.db/person_test, inputFormat:com.twitter.elephantbird.mapred.input.HiveMultiInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:com.twitter.elephantbird.hive.serde.ThriftSerDe, parameters: {serialization.class=com.twitter.elephantbird.examples.thrift.Person, serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), partitionKeys: [FieldSchema(name:part_dt, type:string, comment:null)] , parameters:{EXTERNAL=TRUE, transient_lastDdlTime=1334256942}, viewOriginalText:null, viewExpandedText:null, tableType:EXTERNAL_TABLE) Time taken: 0.147 seconds hive> alter table person_test add partition (part_dt = '20120331T000000Z') location 'hdfs://foo.com/foo/2012/03/31/00'; OK Time taken: 0.149 seconds hive> describe extended person_test partition (part_dt = '20120331T000000Z'); OK part_dt string Detailed Partition Information Partition(values: [20120331T000000Z] , dbName:travis_test, tableName:person_test, createTime:1334257029, lastAccessTime:0, sd:StorageDescriptor(cols: [FieldSchema(name:part_dt, type:string, comment:null)] , location:hdfs://foo.com/foo/2012/03/31/00, inputFormat:com.twitter.elephantbird.mapred.input.HiveMultiInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:com.twitter.elephantbird.hive.serde.ThriftSerDe, parameters:{serialization.class=com.twitter.elephantbird.examples.thrift.Person, serialization.format=1} ), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), parameters: {transient_lastDdlTime=1334257029} ) Time taken: 0.106 seconds hive> PROPOSED BEHAVIOR I believe the correct thing to do is copy the full table schema (serde-reported columns + partition keys) into the partition storage descriptor. Notice the following: Table storage descriptor does not contain any columns, because they are reported by the serde. Partition storage descriptor now contains both the serde-reported schema, and full table schema. hive> create external table travis_test.person_test partitioned by (part_dt string) row format serde "com.twitter.elephantbird.hive.serde.ThriftSerDe" with serdeproperties ("serialization.class"="com.twitter.elephantbird.examples.thrift.Person") stored as inputformat "com.twitter.elephantbird.mapred.input.HiveMultiInputFormat" outputformat "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"; OK Time taken: 0.076 seconds hive> describe extended person_test; OK name struct<first_name:string,last_name:string> from deserializer id int from deserializer email string from deserializer phones array<struct<number:string,type:struct<value:int>>> from deserializer part_dt string Detailed Table Information Table(tableName:person_test, dbName:travis_test, owner:travis, createTime:1334257489, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[], location:hdfs://foo.com/warehouse/travis_test.db/person_test, inputFormat:com.twitter.elephantbird.mapred.input.HiveMultiInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:com.twitter.elephantbird.hive.serde.ThriftSerDe, parameters: {serialization.class=com.twitter.elephantbird.examples.thrift.Person, serialization.format=1} ), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), partitionKeys: [FieldSchema(name:part_dt, type:string, comment:null)] , parameters: {EXTERNAL=TRUE, transient_lastDdlTime=1334257489} , viewOriginalText:null, viewExpandedText:null, tableType:EXTERNAL_TABLE) Time taken: 0.155 seconds hive> alter table person_test add partition (part_dt = '20120331T000000Z') location 'hdfs://foo.com/foo/2012/03/31/00'; OK Time taken: 0.296 seconds hive> describe extended person_test partition (part_dt = '20120331T000000Z'); OK name struct<first_name:string,last_name:string> from deserializer id int from deserializer email string from deserializer phones array<struct<number:string,type:struct<value:int>>> from deserializer part_dt string Detailed Partition Information Partition(values: [20120331T000000Z] , dbName:travis_test, tableName:person_test, createTime:1334257504, lastAccessTime:0, sd:StorageDescriptor(cols: [FieldSchema(name:name, type:struct<first_name:string,last_name:string>, comment:from deserializer), FieldSchema(name:id, type:int, comment:from deserializer), FieldSchema(name:email, type:string, comment:from deserializer), FieldSchema(name:phones, type:array<struct<number:string,type:struct<value:int>>>, comment:from deserializer), FieldSchema(name:part_dt, type:string, comment:null)] , location:hdfs://foo.com/foo/2012/03/31/00, inputFormat:com.twitter.elephantbird.mapred.input.HiveMultiInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:com.twitter.elephantbird.hive.serde.ThriftSerDe, parameters: {serialization.class=com.twitter.elephantbird.examples.thrift.Person, serialization.format =1} ), bucketCols:[], sortCols:[], parameters:{}, primaryRegionName:, secondaryRegions:[]), parameters: {transient_lastDdlTime=1334257504} ) Time taken: 0.133 seconds hive> TEST PLAN empty REVISION DETAIL https://reviews.facebook.net/D2769 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/6297/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
          Hide
          Travis Crawford added a comment -

          Would it be possible for someone to take a look at this patch?

          Show
          Travis Crawford added a comment - Would it be possible for someone to take a look at this patch?
          Hide
          Travis Crawford added a comment -

          Updated the branch against trunk and reran the tests.

          https://travis.ci.cloudbees.com/job/HIVE-2950_partition_full_schema/1/

          There were some test failures in CI so I ran these locally and they passed:

          ant clean package test -Dtestcase=TestCliDriver -Dqfile=create_view_partitioned.q
          ant clean package test -Dtestcase=TestNegativeCliDriver -Dqfile=create_or_replace_view1.q
          ant clean package test -Dtestcase=TestNegativeCliDriver -Dqfile=create_or_replace_view2.q
          ant clean package test -Dtestcase=TestHiveServerSessions
          
          Show
          Travis Crawford added a comment - Updated the branch against trunk and reran the tests. https://travis.ci.cloudbees.com/job/HIVE-2950_partition_full_schema/1/ There were some test failures in CI so I ran these locally and they passed: ant clean package test -Dtestcase=TestCliDriver -Dqfile=create_view_partitioned.q ant clean package test -Dtestcase=TestNegativeCliDriver -Dqfile=create_or_replace_view1.q ant clean package test -Dtestcase=TestNegativeCliDriver -Dqfile=create_or_replace_view2.q ant clean package test -Dtestcase=TestHiveServerSessions
          Hide
          Ashutosh Chauhan added a comment -

          @Travis,
          Can you upload the latest patch at jira. Unfortunately, Phabricator doesn't let you download a patch file.

          Show
          Ashutosh Chauhan added a comment - @Travis, Can you upload the latest patch at jira. Unfortunately, Phabricator doesn't let you download a patch file.
          Hide
          Travis Crawford added a comment -

          The patch is actually the same - the one attached to Jira is up-to-date.

          Show
          Travis Crawford added a comment - The patch is actually the same - the one attached to Jira is up-to-date.
          Hide
          Travis Crawford added a comment -

          Status update:

          Looking into this a bit more, I think we can avoid storing the cols in the metastore if we simply allow partitions to report cols from the serde. Something like this:

          ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java
             public List<FieldSchema> getCols() {
          -    return tPartition.getSd().getCols();
          +    if (SerDeUtils.shouldGetColsFromSerDe(table.getSerializationLib())) {
          +      return table.getCols();
          +    } else {
          +      return tPartition.getSd().getCols();
          +    }
             }
          

          For thrift/protobuf this would work perfectly, since you want all records to have the newest schema, and let thrift/protobuf deal with figuring out missing values, unknown fields, etc.

          Thoughts?

          Show
          Travis Crawford added a comment - Status update: Looking into this a bit more, I think we can avoid storing the cols in the metastore if we simply allow partitions to report cols from the serde. Something like this: ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java public List<FieldSchema> getCols() { - return tPartition.getSd().getCols(); + if (SerDeUtils.shouldGetColsFromSerDe(table.getSerializationLib())) { + return table.getCols(); + } else { + return tPartition.getSd().getCols(); + } } For thrift/protobuf this would work perfectly, since you want all records to have the newest schema, and let thrift/protobuf deal with figuring out missing values, unknown fields, etc. Thoughts?
          Hide
          Ashutosh Chauhan added a comment -

          Travis,
          I liked your first patch better then this one. Semantics is that when you add partition, you store current table's schema in Partition's storage descriptor (which is what your first patch is doing). Your second approach will return the table schema for Partition at the read time, by which time table's schema might have changed. I will test and commit your first patch.

          Show
          Ashutosh Chauhan added a comment - Travis, I liked your first patch better then this one. Semantics is that when you add partition, you store current table's schema in Partition's storage descriptor (which is what your first patch is doing). Your second approach will return the table schema for Partition at the read time, by which time table's schema might have changed. I will test and commit your first patch.
          Hide
          Travis Crawford added a comment -

          In some cases (storing thrift/protobufs) reporting the read-time schema is preferable to the write-time schema. For example, let's say you're storing thrift structs and add a new optional field with default value. In that case all your old records would be automatically upgraded if using the read-time field reporting.

          I played around with this over the weekend and think it could look something like this. If the partition storage descriptor has a serde that you should get the cols from, then do that. Otherwise, use the fields stored in the metastore. Since this is per-partition you could have some partitions using serde-reported fields, and other partitions using hard-coded ones.

          ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java
          -  public List<FieldSchema> getCols() {
          -    return tPartition.getSd().getCols();
          +  public List<FieldSchema> getCols() throws HiveException {
          +    if (SerDeUtils.shouldGetColsFromSerDe(
          +        tPartition.getSd().getSerdeInfo().getSerializationLib())) {
          +      return Hive.getFieldsFromDeserializer(table.getTableName(), getDeserializer());
          +    } else {
          +      return tPartition.getSd().getCols();
          +    }
          

          Thoughts? I still think the original patch (copy table columns into partition storage descriptor at write time) is an improvement, but this dynamic approach would be awesome. Some of our schemas are pretty large so this would save a lot of metastore space

          Show
          Travis Crawford added a comment - In some cases (storing thrift/protobufs) reporting the read-time schema is preferable to the write-time schema. For example, let's say you're storing thrift structs and add a new optional field with default value. In that case all your old records would be automatically upgraded if using the read-time field reporting. I played around with this over the weekend and think it could look something like this. If the partition storage descriptor has a serde that you should get the cols from, then do that. Otherwise, use the fields stored in the metastore. Since this is per-partition you could have some partitions using serde-reported fields, and other partitions using hard-coded ones. ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java - public List<FieldSchema> getCols() { - return tPartition.getSd().getCols(); + public List<FieldSchema> getCols() throws HiveException { + if (SerDeUtils.shouldGetColsFromSerDe( + tPartition.getSd().getSerdeInfo().getSerializationLib())) { + return Hive.getFieldsFromDeserializer(table.getTableName(), getDeserializer()); + } else { + return tPartition.getSd().getCols(); + } Thoughts? I still think the original patch (copy table columns into partition storage descriptor at write time) is an improvement, but this dynamic approach would be awesome. Some of our schemas are pretty large so this would save a lot of metastore space
          Hide
          Travis Crawford added a comment -

          After looking at this further, this change is not actually needed.

          The confusion arises from Hive having two sets of classes to represent the main objects (tables, partitions, ...). If you use metastore.api classes the fields are not available unless stored in the metastore. However, if using the ql.metadata classes, Partition copies the table fields to the partition if they're empty. This works great for thrift-based tables.

          Show
          Travis Crawford added a comment - After looking at this further, this change is not actually needed. The confusion arises from Hive having two sets of classes to represent the main objects (tables, partitions, ...). If you use metastore.api classes the fields are not available unless stored in the metastore. However, if using the ql.metadata classes, Partition copies the table fields to the partition if they're empty. This works great for thrift-based tables.

            People

            • Assignee:
              Travis Crawford
              Reporter:
              Travis Crawford
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development