Details
-
Improvement
-
Status: Closed
-
Trivial
-
Resolution: Fixed
-
0.23.4
-
Reviewed
Description
There are a number of conversions in the Job History Event classes that are totally unnecessary. It appears that they were originally used to convert from the internal avro format, but now many of them do not pull the values from the avro they store them internally.
For example:
/** Get the task type */ public TaskType getTaskType() { return TaskType.valueOf(taskType.toString()); }
The code currently is taking an enum, converting it to a string and then asking the same enum to convert it back to an enum. If java work properly this should be a noop and a reference to the original taskType should be returned.
There are several places that a string is having toString called on it, and since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this point.
Attachments
Attachments
- MAPREDUCE-4822.patch
- 4 kB
- Chu Tong
Activity
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12567924/MAPREDUCE-4822.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3299//console
This message is automatically generated.
No testcase is included as the change is trivial.
For JavaDoc warnings, it is false positive as the same number of warnings are generated on a clean build under my dev environment.
-1 overall.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated 20 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version ) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12567961/MAPREDUCE-4822.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3302//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3302//console
This message is automatically generated.
Hi Chu, thanks for picking this up.
The original intent of the JIRA was to fix more than just the one example occurrence that Bobby pointed out. Looking through the source, I can see more occurrences of unnecessary conversions in MapAttemptFinishedEvent and TaskFinishedEvent.
There's a number of other event classes, like TaskStartedEvent, that are converting back and forth between strings and enums since they store their internal state in a single datum Avro record. We can refactor these to build the datum on-demand like TaskAttemptFinishedEvent does, but that's probably best left to another JIRA. We can keep this one short and simple.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12568110/MAPREDUCE-4822.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3306//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3306//console
This message is automatically generated.
I see there's another unnecessary conversion category that we've missed, and that's the TaskIDs and TaskAttemptIDs. They look like the following:
public TaskAttemptID getAttemptId() { return TaskAttemptID.forName(attemptId.toString()); } .. /** Get task id */ public TaskID getTaskId() { return TaskID.forName(taskid.toString()); } /** Get successful task attempt id */ public TaskAttemptID getSuccessfulTaskAttemptId() { if(successfulAttemptId != null) { return TaskAttemptID.forName(successfulAttemptId.toString()); } return null; }
Sorry I didn't catch this previously.
When I look at the code the TaskAttemptID.forName method does not look like a simple conversion. Therefore, I think it is safer to leave what it is right now.
public static TaskAttemptID forName(String str
) throws IllegalArgumentException {
if(str == null)
return null;
String exceptionMsg = null;
try {
String[] parts = str.split(Character.toString(SEPARATOR));
if(parts.length == 6) {
if(parts[0].equals(ATTEMPT)) {
String type = parts[3];
TaskType t = TaskID.getTaskType(type.charAt(0));
if(t != null)
else
exceptionMsg = "Bad TaskType identifier. TaskAttemptId string : "
+ str + " is not properly formed.";
}
}
} catch (Exception ex)
if (exceptionMsg == null)
{ exceptionMsg = "TaskAttemptId string : " + str + " is not properly formed"; } throw new IllegalArgumentException(exceptionMsg);
}
That code simply parses a string into a TaskAttemptID, the very thing we're trying to avoid doing unnecessarily when we already have the TaskAttemptID. Is there something specific about that conversion you're concerned is necessary when passing an existing TaskAttemptID as a string?
My primary concern is the error checking to throw the exception at the end of the function and I do not know if we want to get ride of that.
The intent of TaskAttemptID.forName() is to be the inverse of its toString(). In these cases we have a valid TaskAttemptID, so the IllegalArgumentException should never be thrown in this context unless forName() or toString() is broken. And I can't imagine why someone would expect and rely on that behavior if it somehow were throwing it. We have an ID, someone is asking for it, why convert it to a string and back before returning it?
To sum up, return TaskAttemptID.forName(taskAttempt.toString()) should just be return taskAttempt. If you're not comfortable making the change, that's fine let me know. We can either file a followup JIRA or I can post the changes to the patch.
Ok, I think your point is more solid. I made the changes as you instructed. Thank you.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12568300/MAPREDUCE-4822.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3312//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3312//console
This message is automatically generated.
+1, lgtm. There's one minor nit where we're now needlessly checking for null:
if(successfulAttemptId != null) { return successfulAttemptId; } return null;
but I'll just fix that as part of the commit.
Integrated in Hadoop-trunk-Commit #3335 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3335/)
MAPREDUCE-4822. Unnecessary conversions in History Events. Contributed by Chu Tong (Revision 1443257)
Result = SUCCESS
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1443257
Files :
- /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptFinishedEvent.java
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java
Thanks, Chu! I committed this to trunk, branch-2, and branch-0.23.
Note that for future JIRAs, the Fix Version field should only be set when the fix is finally committed. The Target Version field should be used when indicating what version(s) a patch is targeting before it is committed.
Integrated in Hadoop-Yarn-trunk #120 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/120/)
MAPREDUCE-4822. Unnecessary conversions in History Events. Contributed by Chu Tong (Revision 1443257)
Result = FAILURE
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1443257
Files :
- /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptFinishedEvent.java
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java
Integrated in Hadoop-Hdfs-0.23-Build #518 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/518/)
svn merge -c 1443257 FIXES: MAPREDUCE-4822. Unnecessary conversions in History Events. Contributed by Chu Tong (Revision 1443271)
Result = SUCCESS
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1443271
Files :
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptFinishedEvent.java
- /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java
Integrated in Hadoop-Hdfs-trunk #1309 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1309/)
MAPREDUCE-4822. Unnecessary conversions in History Events. Contributed by Chu Tong (Revision 1443257)
Result = FAILURE
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1443257
Files :
- /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptFinishedEvent.java
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java
Integrated in Hadoop-Mapreduce-trunk #1337 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1337/)
MAPREDUCE-4822. Unnecessary conversions in History Events. Contributed by Chu Tong (Revision 1443257)
Result = SUCCESS
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1443257
Files :
- /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptFinishedEvent.java
- /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java
Removed unnecessary conversion, please review.