-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Won't Fix
-
Affects Version/s: 1.99.5
-
Fix Version/s: 1.99.5
-
Component/s: None
-
Labels:None
When I export a table that contains date/datetime field from mysql, the job will fail. It happens to all connectors. I've confirmed with HDFS connector and Kite connector.
The log shows
2015-03-07 22:39:08,686 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.joda.time.LocalDate.parse(Ljava/lang/String;)Lorg/joda/time/LocalDate;
at org.apache.sqoop.connector.common.SqoopIDFUtils.toDate(SqoopIDFUtils.java:226)
at org.apache.sqoop.connector.common.SqoopIDFUtils.toObject(SqoopIDFUtils.java:704)
at org.apache.sqoop.connector.common.SqoopIDFUtils.fromCSV(SqoopIDFUtils.java:759)
at org.apache.sqoop.connector.idf.CSVIntermediateDataFormat.getObjectData(CSVIntermediateDataFormat.java:77)
at org.apache.sqoop.job.mr.SqoopMapper$SqoopMapDataWriter.writeContent(SqoopMapper.java:149)
at org.apache.sqoop.job.mr.SqoopMapper$SqoopMapDataWriter.writeArrayRecord(SqoopMapper.java:126)
at org.apache.sqoop.connector.jdbc.GenericJdbcExtractor.extract(GenericJdbcExtractor.java:96)
at org.apache.sqoop.connector.jdbc.GenericJdbcExtractor.extract(GenericJdbcExtractor.java:38)
at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:95)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
The jar joda-time-2.4.jar can be found on /yarn/nm/usercache/root/filecache/[job_id]/.
Update 1: According the job history, joda-time-2.4.jar is on mapreduce.job.classpath.files. The jar is also physically there. I believe the issue is related to some mess classpath configuration. I found an interesting workaround https://vanwilgenburg.wordpress.com/2014/09/07/overriding-hadoop-jars-with-mapreduce-v2-how-to-fix-the-classpath/. After adding job.getConfiguration().set("mapreduce.job.user.classpath.first", "true"); in MapreduceSubmissionEngine.java, the error is gone.
- blocks
-
SQOOP-2035 Sqoop2: KiteConnector From side need to send BigDecimal for decimal type and JODA for date type in Object array
-
- Open
-
-
SQOOP-2037 Sqoop2: HDFSConnector From side need to send BigDecimal for decimal type and JODA for date type in Object array
-
- Open
-
-
SQOOP-2039 Sqoop2: KiteConnector To side need to convert JODA to AvroDate, BigDecimal to Avro Decimal type
-
- Open
-