Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.3.0
-
None
-
None
-
Spark 1.6.0 + Hadoop 2.6.4
-
Important
Description
The equals in this if statement:
if (taskAttemptID.getJobID().equals(jobContext.getJobID()))
is always returning false. The problem is similar to the one described here https://issues.apache.org/jira/browse/MAPREDUCE-1842. The TaskAttemptID.forName method instantiate a new TaskAttemptID object whose type comes from the older API package org.apache.hadoop.mapred. Since the JobContext contains a JobID form the newer package org.apache.hadoop.mapreduce and the equals method shared by the two JobIDs uses this.getClass() == that.getClass, the equals between the two objects will always return false. Comparing the JobID Strings instead of the objects is a possible workaround otherwise another possibility would be to instantiate a org.apache.hadoop.mapreduce.TaskAttemptID object using the one retuned from the TaskAttemptID.forName method.