[HIVE-11129] Issue a warning when copied from UTF-8 to ISO 8859-1

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0, 2.0.0
• Component/s: File Formats
• Assignee: Aihua Xu
• Reporter: Aihua Xu
• Created: 26/Jun/15 19:04
• Updated: 16/Feb/16 23:53
• Resolved: 13/Jul/15 18:48

Description

Copying data from a table using UTF-8 encoding to one using ISO 8859-1 encoding causes data corruption without warning.

CREATE TABLE person_utf8 (name STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
WITH SERDEPROPERTIES ('serialization.encoding'='UTF8');

Put the following data in the table:
Müller,Thomas
Jørgensen,Jørgen
Vega,Andrés
中村,浩人
אביה,נועם

CREATE TABLE person_2 ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
WITH SERDEPROPERTIES ('serialization.encoding'='ISO8859_1')
AS select * from person_utf8;

expected to get mangled data but we should give a warning.