Uploaded image for project: 'Sqoop (Retired)'
  1. Sqoop (Retired)
  2. SQOOP-3065

Sqoop Import fails when column containing unicode data for split-by

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • connectors/sqlserver
    • None

    Description

      sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true --connect jdbc:sqlserver://10.65.141.234;databaseName=sqoop_adapter --password ****** --map-column-java LANGUAGE=String,CODEPAGE=String,C_NAME_JAP_名前=String,C_ADD_GREEK_ΔΙΕΎΘΥΝΣΗ=String,C_MKTSGMT_CHINESE_市場=String,C_COMMENT_KOREAN_덧글=String --username sq_admin --as-avrodatafile --split-by C_MKTSGMT_CHINESE_市場 --columns LANGUAGE,CODEPAGE,C_NAME_JAP_名前,C_ADD_GREEK_ΔΙΕΎΘΥΝΣΗ,C_MKTSGMT_CHINESE_市場,C_COMMENT_KOREAN_덧글 --table 日本語_ΕΛΛΗΝΙΚΑ --target-dir hdfs://asd/blaze/workdir/4133141082177241108/W4133141082177241108_Read_日本語_ΕΛΛΗΝΙΚΑ_m_export_日本語_ΕΛΛΗΝΙΚΑ/staged_files

      overrides a deprecated API.
      2016-12-01 11:50:22 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] Note: Recompile with -Xlint:deprecation for details.
      2016-12-01 11:50:22 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] Writing jar file: /tmp/sqoop-adpqa/compile/9013a77a704c396d503bc9c56a95ba1e/日本語_ΕΛΛΗΝΙΚΑ.jar
      2016-12-01 11:50:22 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] Beginning import of 日本語_ΕΛΛΗΝΙΚΑ
      2016-12-01 11:50:22 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] mapred.jar is deprecated. Instead, use mapreduce.job.jar
      2016-12-01 11:50:23 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] Executing SQL statement: SELECT t.* FROM [日本語_ΕΛΛΗΝΙΚΑ] AS t WHERE 1=0
      2016-12-01 11:50:23 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] Executing SQL statement: SELECT t.* FROM [日本語_ΕΛΛΗΝΙΚΑ] AS t WHERE 1=0
      2016-12-01 11:50:23 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] Writing Avro schema file: /tmp/sqoop-adpqa/compile/9013a77a704c396d503bc9c56a95ba1e/日本語_ΕΛΛΗΝΙΚΑ.avsc
      2016-12-01 11:50:23 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
      2016-12-01 11:50:23 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] SQOOP_HOME is unset. May not be able to find all job dependencies.
      2016-12-01 11:50:24 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] PriviledgedActionException as:adpqa (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
      2016-12-01 11:50:24 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
      2016-12-01 11:50:24 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] PriviledgedActionException as:adpqa (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
      2016-12-01 11:50:24 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] Failing over to rm71
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] Using read commited transaction isolation
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] BoundingValsQuery: SELECT MIN([C_MKTSGMT_CHINESE_市場]), MAX([C_MKTSGMT_CHINESE_市場]) FROM [日本語_ΕΛΛΗΝΙΚΑ]
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] Generating splits for a textual index column.
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] If your database sorts in a case-insensitive order, this may result in a partial import or duplicate records.
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] You are strongly encouraged to choose an integral split column.
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] Cleaning up the staging area /tmp/hadoop-yarn/staging/adpqa/.staging/job_1480054223215_0084
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] PriviledgedActionException as:adpqa (auth:PROXY) via adpqa (auth:KERBEROS) cause:java.io.IOException: 0.0015106201171875 is greater than 0.001205468783541194110069724588674860456249948725826470763422548770904541015625
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] Encountered IOException running import job: java.io.IOException: 0.0015106201171875 is greater than 0.001205468783541194110069724588674860456249948725826470763422548770904541015625
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:204)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:305)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:322)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:200)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at java.security.AccessController.doPrivileged(Native Method)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at javax.security.auth.Subject.doAs(Subject.java:422)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at java.security.AccessController.doPrivileged(Native Method)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at javax.security.auth.Subject.doAs(Subject.java:422)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1325)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:203)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:176)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:273)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:692)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.sqoop.manager.SQLServerManager.importTable(SQLServerManager.java:163)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:507)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at java.lang.reflect.Method.invoke(Method.java:497)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at com.infa.adapter.sqoop.client.Client.invoke(Client.java:86)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at com.infa.adapter.sqoop.client.Client.main(Client.java:40)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at java.lang.reflect.Method.invoke(Method.java:497)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] Caused by: 0.0015106201171875 is greater than 0.001205468783541194110069724588674860456249948725826470763422548770904541015625
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.sqoop.mapreduce.db.TextSplitter.split(TextSplitter.java:157)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.sqoop.mapreduce.db.TextSplitter.split(TextSplitter.java:116)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:201)
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762] ... 33 more
      2016-12-01 11:50:25 <TASK_140221035255552-MAPPING> INFO: [APPSDK_Msg_1762]

      Thanks & Regards,
      Rohith

      Attachments

        Activity

          People

            Unassigned Unassigned
            senthils Senthil Ganesh
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: