Details
-
Bug
-
Status: Open
-
Blocker
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
Hi,
We are using Sqoop version 1.4.6-cdh5.8.5 for importing numeric data types from Oracle to Avro Decimal Logical type (Bytes as Decimal with precision and scale)
The import works ok except for cases in which the values stored in Oracle have the scale lower than the one defined in table ddl.
Ex. the column is defined as NUMERIC(20,2) and the value stored is "3.2" (only one digit after decimal point); in this case we receive the error message:
Error: org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.AvroTypeException: Cannot encode decimal with scale 1 as scale 2
This error was discussed in https://issues.apache.org/jira/browse/AVRO-1864 with the resolution that this sqoop behavior is correct (it cannot add by default an extra info for transforming "3.2" to "3.20")
We tried below methods for conversion to AVRO Decimal(20,2) Logical datatype:
1). use --map-column-java "COLNAME=java.math.BigDecimal" in sqoop import command
error received:
Error: org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.AvroTypeException: Cannot encode decimal with scale 1 as scale 2
2). use --map-column-java "COL1=Decimal(20%2C2)" in sqoop command
error received:
ERROR tool.ImportTool: Import failed: No ResultSet method for Java type Decimal(20,2)
3). made the column as varchar2(100) in Oracle database, stored the value as "3.20" and use _--map-column-java "COLNAME=java.math.BigDecimal" _in sqoop command
error received:
ERROR tool.ImportTool: Import failed: Cannot convert to AVRO type java.math.BigDecimal
There is any way in which we can instruct sqoop to map the column from Oracle table to AVRO Decimal logical type (precision=20, scale=2) and import the "3.2" value from Oracle database as "3.20" type Decimal(20,2) in AVRO file?
Thanks,
Liviu