[HIVE-18083] Support UTF8 in MySQL Metastore Backend

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.0, 3.0.0
• Component/s: Metastore, Standalone Metastore
• Assignee: Unassigned
• Reporter: David Mollitor
• Created: 16/Nov/17 13:55
• Updated: 31/Aug/20 16:46

Description

hive-schema-2.2.0.mysql.sql

CREATE TABLE IF NOT EXISTS `COLUMNS_V2` (
  `CD_ID` bigint(20) NOT NULL,
  `COMMENT` varchar(256) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
  `COLUMN_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
  `TYPE_NAME` varchar(4000) DEFAULT NULL,
  `INTEGER_IDX` int(11) NOT NULL,
  PRIMARY KEY (`CD_ID`,`COLUMN_NAME`),
  KEY `COLUMNS_V2_N49` (`CD_ID`),
  CONSTRAINT `COLUMNS_V2_FK1` FOREIGN KEY (`CD_ID`) REFERENCES `CDS` (`CD_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Hive explicitly defines a CHARACTER SET latin1 COLLATE latin1_bin in the schema design. This explicit definition should either be removed, so that it can fallback onto the database administrator's defaults, or changed to CHARACTER SET utf8 COLLATE utf8_bin to change the explicit definition to utf8.

This will allow Hive to support UTF8 characters in MySQL backend databases for our international friends.

Issue Links

• is related to: HIVE-3914 use Chinese in hive column comment and table comment
• is related to: HIVE-14156 Problem with Chinese characters as partition value when using MySQL
• is related to: HIVE-16558 Use mysql to store hive metadata，support chinese comments for column
• is related to: HIVE-5962 Support UTF8 character for table or column names
• relates to: HIVE-12157 Support unicode for table/column names
• relates to: IMPALA-2019 Proper UTF-8 support in string functions
• relates to: HIVE-23149 Consistency of Parsing Object Identifiers