[SQOOP-3094] Add (import + --as-avrodatafile) with Oracle BINARY_DOUBLE

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Markus Kemper
• Created: 27/Dec/16 18:42
• Updated: 27/Dec/16 18:50

Description

Some users are not able to easily implement the Sqoop option (--map-column-java, Example 1 below).

The ask here is to find a way to natively support the Oracle BINARY_DOUBLE type in Sqoop natively without using --map-column-java (see Example 2 below)

Example 1

sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table T1 --target-dir /user/user1/t1 --delete-target-dir --verbose --num-mappers 1 --as-avrodatafile --map-column-java C2=String
hdfs dfs -ls /user/user1/t1/*.avro
avro-tools tojson --pretty 'hdfs://namenode.cloudera.com/user/user1/t1/part-m-00000.avro'

Output:
16/12/27 10:28:03 INFO mapreduce.ImportJobBase: Transferred 320 bytes in 39.1563 seconds (8.1724 bytes/sec)
16/12/27 10:28:03 INFO mapreduce.ImportJobBase: Retrieved 1 records.
---
-rw-r--r--   3 user1 user1        320 2016-12-27 10:28 /user/user1/t1/part-m-00000.avro
---
{
  "C1" : {
    "string" : "1"
  },
  "C2" : {
    "string" : "1.1"
  }
}

Example 2

sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table T1 --target-dir /user/user1/t1 --delete-target-dir --verbose --num-mappers 1 --as-parquetfile 

Output:
16/12/27 10:05:43 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM T1 t WHERE 1=0
16/12/27 10:05:43 DEBUG manager.SqlManager: Found column C1 of type [2, 38, 0]
16/12/27 10:05:43 DEBUG manager.SqlManager: Found column C2 of type [101, 0, 0]
16/12/27 10:05:43 DEBUG util.ClassLoaderStack: Restoring classloader: java.net.FactoryURLClassLoader@55465b1f
16/12/27 10:05:43 ERROR tool.ImportTool: Imported Failed: Cannot convert SQL type 101

sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table T1 --target-dir /user/user1/t1 --delete-target-dir --verbose --num-mappers 2 --as-parquetfile  --direct

Output:
16/12/27 10:08:50 DEBUG oracle.OraOopUtilities: The Oracle table context has been derived from:
	oracleConnectionUserName = sqoop
	tableStr = T1
	as:
	owner : SQOOP
	table : T1
16/12/27 10:08:50 INFO oracle.OraOopManagerFactory: 
**************************************************
*** Using Data Connector for Oracle and Hadoop ***
**************************************************
<SNIP>
16/12/27 10:08:53 INFO manager.SqlManager: Executing SQL statement: SELECT C1,C2 FROM T1 WHERE 1=0
16/12/27 10:08:53 DEBUG manager.SqlManager: Found column C1
16/12/27 10:08:53 DEBUG manager.SqlManager: Found column C2
16/12/27 10:08:53 DEBUG util.ClassLoaderStack: Restoring classloader: java.net.FactoryURLClassLoader@7e087bf5
16/12/27 10:08:53 ERROR tool.ImportTool: Imported Failed: Cannot convert SQL type 101

Issue Links

• is related to: SQOOP-3089 Sqoop import + --as-parquetfile + Oracle + BINARY_DOUBLE fails with insufficient message
• is related to: SQOOP-3095 Add (import + --as-parquetfile) with Oracle BINARY_DOUBLE