Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Critical
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.8.0, 2.7.2, 2.6.3, 3.0.0-alpha1
-
Component/s: jobhistoryserver
-
Labels:None
-
Target Version/s:
-
Hadoop Flags:Reviewed
Description
We meet memory leak issues for JHS in a large cluster which is caused by code below doesn't release FSDataInputStream in exception case. MAPREDUCE-6273 should fix most cases that exceptions get thrown. However, we still need to fix the memory leak for occasional case.
private String getJobSummary(FileContext fc, Path path) throws IOException { Path qPath = fc.makeQualified(path); FSDataInputStream in = fc.open(qPath); String jobSummaryString = in.readUTF(); in.close(); return jobSummaryString; }
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 7s | docker + precommit patch detected. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | mvninstall | 3m 15s | trunk passed |
| +1 | compile | 0m 14s | trunk passed with JDK v1.8.0_60 |
| +1 | compile | 0m 14s | trunk passed with JDK v1.7.0_79 |
| +1 | checkstyle | 0m 8s | trunk passed |
| +1 | mvneclipse | 0m 14s | trunk passed |
| +1 | findbugs | 0m 34s | trunk passed |
| +1 | javadoc | 0m 13s | trunk passed with JDK v1.8.0_60 |
| +1 | javadoc | 0m 15s | trunk passed with JDK v1.7.0_79 |
| +1 | mvninstall | 0m 21s | the patch passed |
| +1 | compile | 0m 15s | the patch passed with JDK v1.8.0_60 |
| +1 | javac | 0m 15s | the patch passed |
| +1 | compile | 0m 17s | the patch passed with JDK v1.7.0_79 |
| +1 | javac | 0m 17s | the patch passed |
| +1 | checkstyle | 0m 8s | the patch passed |
| +1 | mvneclipse | 0m 16s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 0m 44s | the patch passed |
| +1 | javadoc | 0m 16s | the patch passed with JDK v1.8.0_60 |
| +1 | javadoc | 0m 18s | the patch passed with JDK v1.7.0_79 |
| +1 | unit | 6m 21s | hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_60. |
| +1 | unit | 5m 58s | hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_79. |
| -1 | asflicense | 0m 20s | Patch generated 14 ASF License warnings. |
| 21m 37s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-29 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12769533/MAPREDUCE-6528.patch |
| JIRA Issue | |
| Optional Tests | asflicense javac javadoc mvninstall unit findbugs checkstyle compile |
| uname | Linux f71f51e42104 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/patchprocess/apache-yetus-67f42f1/precommit/personality/hadoop.sh |
| git revision | trunk / c416999 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 |
| findbugs | v3.0.0 |
| JDK v1.7.0_79 Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6098/testReport/ |
| asflicense | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6098/artifact/patchprocess/patch-asflicense-problems.txt |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs |
| Max memory used | 224MB |
| Powered by | Apache Yetus http://yetus.apache.org |
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6098/console |
This message was automatically generated.
Junping Du Nice catch..Thanks for reporting this..can you use try-with-resources..?
Thanks for reporting this..can you use try-with-resources..?
Given code with final block is already there. Any advantage for try-with-resources?
I'd like to get this put in 2.6.3 too, so it'd be great if we can make this work across JDKs. Thanks.
Good point, Vinod! Let's keep the patch as it is now as try-with-resources won't be supported in earlier version of JDKs.
Agreed..Since it's planned for 2.6.3 release..Current patch LGTM..+ 1 (nonbinding)..
Thanks Brahma Reddy Battula! Can someone commit this patch in? It is quite straight-forward.
+1, committing this.
Thanks to Junping for the contribution and to Brahma for additional review! I committed this to trunk, branch-2, branch-2.7, and branch-2.6.
FAILURE: Integrated in Hadoop-trunk-Commit #8732 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8732/)
MAPREDUCE-6528. Memory leak for HistoryFileManager.getJobSummary(). (jlowe: rev 6344b6a7694c70f296392b6462dba452ff762109)
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
- hadoop-mapreduce-project/CHANGES.txt
Thanks Jason to review and commit this in!
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #607 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/607/)
MAPREDUCE-6528. Memory leak for HistoryFileManager.getJobSummary(). (jlowe: rev 6344b6a7694c70f296392b6462dba452ff762109)
- hadoop-mapreduce-project/CHANGES.txt
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2550 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2550/)
MAPREDUCE-6528. Memory leak for HistoryFileManager.getJobSummary(). (jlowe: rev 6344b6a7694c70f296392b6462dba452ff762109)
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
- hadoop-mapreduce-project/CHANGES.txt
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #620 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/620/)
MAPREDUCE-6528. Memory leak for HistoryFileManager.getJobSummary(). (jlowe: rev 6344b6a7694c70f296392b6462dba452ff762109)
- hadoop-mapreduce-project/CHANGES.txt
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
FAILURE: Integrated in Hadoop-Yarn-trunk #1343 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1343/)
MAPREDUCE-6528. Memory leak for HistoryFileManager.getJobSummary(). (jlowe: rev 6344b6a7694c70f296392b6462dba452ff762109)
- hadoop-mapreduce-project/CHANGES.txt
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #556 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/556/)
MAPREDUCE-6528. Memory leak for HistoryFileManager.getJobSummary(). (jlowe: rev 6344b6a7694c70f296392b6462dba452ff762109)
- hadoop-mapreduce-project/CHANGES.txt
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
FAILURE: Integrated in Hadoop-Hdfs-trunk #2494 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2494/)
MAPREDUCE-6528. Memory leak for HistoryFileManager.getJobSummary(). (jlowe: rev 6344b6a7694c70f296392b6462dba452ff762109)
- hadoop-mapreduce-project/CHANGES.txt
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
Attach a simple patch to fix it. The fix is quite straightforward, so no need for unit test.