Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-3677

Encoding Issue - ISO-8859-1

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.8.1
    • Fix Version/s: 0.8.1
    • Labels:
      None
    • Environment:

      Amazon EMR with Hive (Hive 0.8.1 and haddop 1.0.3)

      Description

      We’ve created a very simple example using Amazon EMR with Hive which is basically create a single table with Hive and load some data inside this table. Below you’ll find the code that has been used:

      //CREATE TABLE CODE

      CREATE TABLE sampletable (
      valorstring STRING, valordecimal DOUBLE)
      ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe'
      WITH SERDEPROPERTIES (
      'serialization.format'='org.apache.hadoop.hive.serde2.thrift.TCTLSeparatedProtocol',
      'quote.delim'='("|\\[|
      ])',
      'field.delim'=' ',
      'serialization.null.format'='-')
      STORED AS TEXTFILE;

      //LOAD DATA CODE
      LOAD DATA LOCAL INPATH '/tmp/sampletable.txt' OVERWRITE INTO TABLE sampletable;

      Here is the text file content that we are using to load the data:

      /tmp/sampletable.txt
      "Exemplo de texto com acentuação" 90,15
      "Exemplo de texto com acentuação" 80.15

      The problem that we are facing seems to be with the enconding that is been used in Hive configuration. Seems to me that it is been used UTF-8 but for Brazilian format we’ll need to use ISO-8859-1.

      In the example above, when the data is loaded inside the table and we perform a simple select (Select * from sampletable) the text with accentuation is returned totally wrong and the double value with comma is returned as null.
      We’ve already changed the variable LANG in enviroment and Hive variables with SET, but it doesn’t work so far.

      Thank you in advance!!!

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              sergekm Sergio Kameoka
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: