[SQOOP-2189] Sqoop2: Failed to export date column

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.99.5
• Fix Version/s: 1.99.5
• Assignee: Qian Xu
• Reporter: Qian Xu
• Created: 07/Mar/15 14:43
• Updated: 13/Mar/15 03:30
• Resolved: 08/Mar/15 17:17

Description

When I export a table that contains date/datetime field from mysql, the job will fail. It happens to all connectors. I've confirmed with HDFS connector and Kite connector.

The log shows

2015-03-07 22:39:08,686 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.joda.time.LocalDate.parse(Ljava/lang/String;)Lorg/joda/time/LocalDate;
	at org.apache.sqoop.connector.common.SqoopIDFUtils.toDate(SqoopIDFUtils.java:226)
	at org.apache.sqoop.connector.common.SqoopIDFUtils.toObject(SqoopIDFUtils.java:704)
	at org.apache.sqoop.connector.common.SqoopIDFUtils.fromCSV(SqoopIDFUtils.java:759)
	at org.apache.sqoop.connector.idf.CSVIntermediateDataFormat.getObjectData(CSVIntermediateDataFormat.java:77)
	at org.apache.sqoop.job.mr.SqoopMapper$SqoopMapDataWriter.writeContent(SqoopMapper.java:149)
	at org.apache.sqoop.job.mr.SqoopMapper$SqoopMapDataWriter.writeArrayRecord(SqoopMapper.java:126)
	at org.apache.sqoop.connector.jdbc.GenericJdbcExtractor.extract(GenericJdbcExtractor.java:96)
	at org.apache.sqoop.connector.jdbc.GenericJdbcExtractor.extract(GenericJdbcExtractor.java:38)
	at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:95)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

The jar joda-time-2.4.jar can be found on /yarn/nm/usercache/root/filecache/[job_id]/.

Update 1: According the job history, joda-time-2.4.jar is on mapreduce.job.classpath.files. The jar is also physically there. I believe the issue is related to some mess classpath configuration. I found an interesting workaround https://vanwilgenburg.wordpress.com/2014/09/07/overriding-hadoop-jars-with-mapreduce-v2-how-to-fix-the-classpath/. After adding job.getConfiguration().set("mapreduce.job.user.classpath.first", "true"); in MapreduceSubmissionEngine.java, the error is gone.

Issue Links

• blocks: SQOOP-2035 Sqoop2: KiteConnector From side need to send BigDecimal for decimal type and JODA for date type in Object array
• blocks: SQOOP-2037 Sqoop2: HDFSConnector From side need to send BigDecimal for decimal type and JODA for date type in Object array
• blocks: SQOOP-2039 Sqoop2: KiteConnector To side need to convert JODA to AvroDate, BigDecimal to Avro Decimal type