Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-2189

Sqoop2: Failed to export date column

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 1.99.5
    • Fix Version/s: 1.99.5
    • Component/s: None
    • Labels:
      None

      Description

      When I export a table that contains date/datetime field from mysql, the job will fail. It happens to all connectors. I've confirmed with HDFS connector and Kite connector.

      The log shows

      2015-03-07 22:39:08,686 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.joda.time.LocalDate.parse(Ljava/lang/String;)Lorg/joda/time/LocalDate;
      	at org.apache.sqoop.connector.common.SqoopIDFUtils.toDate(SqoopIDFUtils.java:226)
      	at org.apache.sqoop.connector.common.SqoopIDFUtils.toObject(SqoopIDFUtils.java:704)
      	at org.apache.sqoop.connector.common.SqoopIDFUtils.fromCSV(SqoopIDFUtils.java:759)
      	at org.apache.sqoop.connector.idf.CSVIntermediateDataFormat.getObjectData(CSVIntermediateDataFormat.java:77)
      	at org.apache.sqoop.job.mr.SqoopMapper$SqoopMapDataWriter.writeContent(SqoopMapper.java:149)
      	at org.apache.sqoop.job.mr.SqoopMapper$SqoopMapDataWriter.writeArrayRecord(SqoopMapper.java:126)
      	at org.apache.sqoop.connector.jdbc.GenericJdbcExtractor.extract(GenericJdbcExtractor.java:96)
      	at org.apache.sqoop.connector.jdbc.GenericJdbcExtractor.extract(GenericJdbcExtractor.java:38)
      	at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:95)
      	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
      	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
      	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
      

      The jar joda-time-2.4.jar can be found on /yarn/nm/usercache/root/filecache/[job_id]/.

      Update 1: According the job history, joda-time-2.4.jar is on mapreduce.job.classpath.files. The jar is also physically there. I believe the issue is related to some mess classpath configuration. I found an interesting workaround https://vanwilgenburg.wordpress.com/2014/09/07/overriding-hadoop-jars-with-mapreduce-v2-how-to-fix-the-classpath/. After adding job.getConfiguration().set("mapreduce.job.user.classpath.first", "true"); in MapreduceSubmissionEngine.java, the error is gone.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stanleyxu2005 Qian Xu
                Reporter:
                stanleyxu2005 Qian Xu
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: