Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
Some users are not able to easily implement the Sqoop option (--map-column-java, Example 1 below).
The ask here is to find a way to natively support the Oracle BINARY_DOUBLE type in Sqoop natively without using --map-column-java (see Example 2 below)
Example 1
sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table T1 --target-dir /user/user1/t1 --delete-target-dir --verbose --num-mappers 1 --as-parquetfile --map-column-java C2=String hdfs dfs -ls /user/user1/t1/*.parquet parquet-tools cat --json 'hdfs://namenode.cloudera.com/user/user1/t1/9c4742a1-68e2-4fd7-8264-6e84110ac409.parquet' Output: 16/12/27 10:34:05 INFO mapreduce.ImportJobBase: Transferred 1.5537 KB in 42.1369 seconds (37.7579 bytes/sec) 16/12/27 10:34:05 INFO mapreduce.ImportJobBase: Retrieved 1 records. --- -rw-r--r-- 3 user1 user1 565 2016-12-27 10:34 /user/user1/t1/9c4742a1-68e2-4fd7-8264-6e84110ac409.parquet --- {"C1":"1","C2":"1.1"}
Example 2
sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table T1 --target-dir /user/user1/t1 --delete-target-dir --verbose --num-mappers 1 --as-parquetfile Output: 16/12/27 10:05:43 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM T1 t WHERE 1=0 16/12/27 10:05:43 DEBUG manager.SqlManager: Found column C1 of type [2, 38, 0] 16/12/27 10:05:43 DEBUG manager.SqlManager: Found column C2 of type [101, 0, 0] 16/12/27 10:05:43 DEBUG util.ClassLoaderStack: Restoring classloader: java.net.FactoryURLClassLoader@55465b1f 16/12/27 10:05:43 ERROR tool.ImportTool: Imported Failed: Cannot convert SQL type 101 sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table T1 --target-dir /user/user1/t1 --delete-target-dir --verbose --num-mappers 2 --as-parquetfile --direct Output: 16/12/27 10:08:50 DEBUG oracle.OraOopUtilities: The Oracle table context has been derived from: oracleConnectionUserName = sqoop tableStr = T1 as: owner : SQOOP table : T1 16/12/27 10:08:50 INFO oracle.OraOopManagerFactory: ************************************************** *** Using Data Connector for Oracle and Hadoop *** ************************************************** <SNIP> 16/12/27 10:08:53 INFO manager.SqlManager: Executing SQL statement: SELECT C1,C2 FROM T1 WHERE 1=0 16/12/27 10:08:53 DEBUG manager.SqlManager: Found column C1 16/12/27 10:08:53 DEBUG manager.SqlManager: Found column C2 16/12/27 10:08:53 DEBUG util.ClassLoaderStack: Restoring classloader: java.net.FactoryURLClassLoader@7e087bf5 16/12/27 10:08:53 ERROR tool.ImportTool: Imported Failed: Cannot convert SQL type 101
Attachments
Issue Links
- relates to
-
SQOOP-3089 Sqoop import + --as-parquetfile + Oracle + BINARY_DOUBLE fails with insufficient message
- Open
-
SQOOP-3094 Add (import + --as-avrodatafile) with Oracle BINARY_DOUBLE
- Open