[SQOOP-3025] Oracle Direct export from HCatalog parquet table gives NPE in kerberized cluster

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.4.6
• Component/s: build, codegen, connectors/oracle, hive-integration, metastore

Environment

Hadoop 2.6; Sqoop 1.4.6; Oracle 12c

• Assignee: Unassigned
• Reporter: Ruslan Dautkhanov
• Created: 15/Oct/16 16:24
• Updated: 19/Oct/16 03:12
• Resolved: 18/Oct/16 20:16

Description

Getting following exception in one of the mappers stack when try to export to Oracle from Hive Parquet

2016-10-14 23:14:01,027 ERROR [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: The following error occurred during configurePreparedStatement()
java.lang.NullPointerException
	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.setBindValueAtName(OraOopOutputFormatBase.java:432)
	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.configurePreparedStatementColumns(OraOopOutputFormatBase.java:547)
	at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:246)
	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303)
	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153)
	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194)
	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578)
	at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107)
	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
	at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
2016-10-14 23:14:01,074 INFO [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Time spent performing an "exchange subpartition with table": 0.046365 sec.
2016-10-14 23:14:01,074 DEBUG [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Dropping temporary mapper table "ORAOOP_20161014_231320_3"
2016-10-14 23:14:01,131 INFO [main] org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector@54848551
java.io.IOException: java.sql.SQLException: java.lang.NullPointerException
	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:197)
	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578)
	at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107)
	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
	at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.sql.SQLException: java.lang.NullPointerException
	at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:255)
	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303)
	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153)
	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194)

Will attach full log file as an attachment.

A minimal sqoop export parameters that reproduce this problem:

sqoop export -Dmapred.map.max.attempts=1 --connect "jdbc:oracle:thin:@//somehost:1355/adash_prod_batch" --username DISCOVER_STG_OWNER --password somePassword --table amf_trans_stg --hcatalog-database disc_dv --hcatalog-table amf_trans_dv_09142016 --num-mappers 2 --columns "individ,part_code" --direct --verbose

Issue Links

• is broken by: SQOOP-3030 Export to Oracle using (--direct + --hcatalog + --columns) with columns having the wrong case sensitivity fails with NPE
• is related to: SQOOP-3029 Add an option for uppercase/lowercase column name mapping between HCatalog and RDBMS cloumn name list