[HIVE-3677] Encoding Issue - ISO-8859-1

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.8.1
• Component/s: Configuration, Import/Export

Environment

Amazon EMR with Hive (Hive 0.8.1 and haddop 1.0.3)

• Assignee: Unassigned
• Reporter: Sergio Kameoka
• Created: 06/Nov/12 18:26
• Updated: 21/Oct/22 07:20

Description

We’ve created a very simple example using Amazon EMR with Hive which is basically create a single table with Hive and load some data inside this table. Below you’ll find the code that has been used:

//CREATE TABLE CODE

CREATE TABLE sampletable (
valorstring STRING, valordecimal DOUBLE)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe'
WITH SERDEPROPERTIES (
'serialization.format'='org.apache.hadoop.hive.serde2.thrift.TCTLSeparatedProtocol',
'quote.delim'='("|\\[|
])',
'field.delim'=' ',
'serialization.null.format'='-')
STORED AS TEXTFILE;

//LOAD DATA CODE
LOAD DATA LOCAL INPATH '/tmp/sampletable.txt' OVERWRITE INTO TABLE sampletable;

Here is the text file content that we are using to load the data:

/tmp/sampletable.txt
"Exemplo de texto com acentuação" 90,15
"Exemplo de texto com acentuação" 80.15

The problem that we are facing seems to be with the enconding that is been used in Hive configuration. Seems to me that it is been used UTF-8 but for Brazilian format we’ll need to use ISO-8859-1.

In the example above, when the data is loaded inside the table and we perform a simple select (Select * from sampletable) the text with accentuation is returned totally wrong and the double value with comma is returned as null.
We’ve already changed the variable LANG in enviroment and Hive variables with SET, but it doesn’t work so far.

Thank you in advance!!!