Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-3025

Oracle Direct export from HCatalog parquet table gives NPE in kerberized cluster

    XMLWordPrintableJSON

    Details

      Description

      Getting following exception in one of the mappers stack when try to export to Oracle from Hive Parquet

      2016-10-14 23:14:01,027 ERROR [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: The following error occurred during configurePreparedStatement()
      java.lang.NullPointerException
      	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.setBindValueAtName(OraOopOutputFormatBase.java:432)
      	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.configurePreparedStatementColumns(OraOopOutputFormatBase.java:547)
      	at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:246)
      	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303)
      	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153)
      	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194)
      	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578)
      	at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107)
      	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
      	at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
      	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
      	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
      	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
      2016-10-14 23:14:01,074 INFO [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Time spent performing an "exchange subpartition with table": 0.046365 sec.
      2016-10-14 23:14:01,074 DEBUG [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Dropping temporary mapper table "ORAOOP_20161014_231320_3"
      2016-10-14 23:14:01,131 INFO [main] org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector@54848551
      java.io.IOException: java.sql.SQLException: java.lang.NullPointerException
      	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:197)
      	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578)
      	at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107)
      	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
      	at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
      	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
      	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
      	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
      Caused by: java.sql.SQLException: java.lang.NullPointerException
      	at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:255)
      	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303)
      	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153)
      	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194)
      

      Will attach full log file as an attachment.

      A minimal sqoop export parameters that reproduce this problem:

      sqoop export -Dmapred.map.max.attempts=1 --connect "jdbc:oracle:thin:@//somehost:1355/adash_prod_batch" --username DISCOVER_STG_OWNER --password somePassword --table amf_trans_stg --hcatalog-database disc_dv --hcatalog-table amf_trans_dv_09142016 --num-mappers 2 --columns "individ,part_code" --direct --verbose

        Attachments

        1. scoop_output.log
          33 kB
          Ruslan Dautkhanov
        2. sqoop_OracleDirect_mapper_nullPointer_exception.txt
          18 kB
          Ruslan Dautkhanov

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Tagar Ruslan Dautkhanov
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: