Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-3285

Hcatalog import is loosing precision on DECIMAL fields

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.4.7
    • Fix Version/s: None
    • Component/s: hive-integration
    • Labels:

      Description

      Import with hcatalog for DECIMAL fields is loosing precision.

      The bug is in org.apache.sqoop.mapreduce.hcat.SqoopHCatImportHelper.convertNumberTypes()

      Internally the BigDecimal value is converted to a Double then back into a BigDecimal.

      As a test I imported from a MySQL db a DECIMAL(38,8) field:

      create table sampledata1 ( n38 DECIMAL(38,8) );
      insert into sampledata1 ( n38 ) values ( -123456789012345678901234567890.12345678 )
      
      sqoop import \
        --connect jdbc:mysql://10.210.144.22:3306/greg --username greg -P \
        -m 1 \
        --table sampledata1 \
        --hcatalog-table sampledata1_orc2 \
        --create-hcatalog-table \
        --hcatalog-storage-stanza 'stored as orc'
      
      
      

      The result in the hive table does not match the input.

      Original:
      -123456789012345678901234567890.12345678
      Result:
      -123456789012345677877719597056.00000000

      I get the same result with Orc, RCfile, and Avro files.

       

      I believe I found the problem and will provide a patch shortly.

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              glindholm Greg Lindholm
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: