[HIVE-1364] Increase the maximum length of various metastore fields, and remove TYPE_NAME from COLUMNS primary key

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.5.0
• Fix Version/s: 0.6.0
• Component/s: Metastore
• Hadoop Flags: Reviewed
• Tags: mysql serdeproperties jpox length
• Assignee: Carl Steinbach
• Reporter: Carl Steinbach
• Created: 24/May/10 16:32
• Updated: 27/Oct/15 16:27
• Resolved: 06/Oct/10 20:43

Description

The value component of a SERDEPROPERTIES key/value pair is currently limited
to a maximum length of 767 characters. I believe that the motivation for limiting the length to
767 characters is that this value is the maximum allowed length of an index in
a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315

• The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
• The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
• We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.

I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.

Issue Links

• blocks: HIVE-1427 Provide metastore schema migration scripts (0.5 -> 0.6)
• is depended upon by: HIVE-1632 Column length not sufficient for large STRUCT definitions
• relates to: HIVE-1426 Programatically enforce Metastore property length limits
• relates to: HIVE-1427 Provide metastore schema migration scripts (0.5 -> 0.6)