Uploaded image for project: 'Apache Blur'
  1. Apache Blur
  2. BLUR-449

The commitOrAbortJob method of the BlurOutputCommitter class ends up always with the warning "TaskAttempt JobID [{0}] does not match JobContext JobId [{1}]"

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.3.0
    • None
    • Blur MapReduce
    • None
    • Spark 1.6.0 + Hadoop 2.6.4
    • Important

    Description

      The equals in this if statement:

      if (taskAttemptID.getJobID().equals(jobContext.getJobID()))

      is always returning false. The problem is similar to the one described here https://issues.apache.org/jira/browse/MAPREDUCE-1842. The TaskAttemptID.forName method instantiate a new TaskAttemptID object whose type comes from the older API package org.apache.hadoop.mapred. Since the JobContext contains a JobID form the newer package org.apache.hadoop.mapreduce and the equals method shared by the two JobIDs uses this.getClass() == that.getClass, the equals between the two objects will always return false. Comparing the JobID Strings instead of the objects is a possible workaround otherwise another possibility would be to instantiate a org.apache.hadoop.mapreduce.TaskAttemptID object using the one retuned from the TaskAttemptID.forName method.

      Attachments

        Activity

          There are no comments yet on this issue.

          People

            Unassigned Unassigned
            max_87 Massimiliano Mattetti

            Dates

              Created:
              Updated:

              Slack

                Issue deployment