Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2855

ResourceBundle lookup during counter name resolution takes a lot of time

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.2
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time.
    • Target Version/s:

      Description

      Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.

      1. MR2855.txt
        1 kB
        Siddharth Seth

        Issue Links

          Activity

          Karthik Kambatla made changes -
          Link This issue is cloned as MAPREDUCE-4565 [ MAPREDUCE-4565 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1004 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1004/)
          MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. (Revision 1294436)

          Result = SUCCESS
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294436
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1004 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1004/ ) MAPREDUCE-2855 . Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. (Revision 1294436) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294436 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #210 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/210/)
          MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth.
          svn merge --ignore-ancestry -c 1294436 ../../trunk (Revision 1294437)

          Result = FAILURE
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294437
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #210 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/210/ ) MAPREDUCE-2855 . Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. svn merge --ignore-ancestry -c 1294436 ../../trunk (Revision 1294437) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294437 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #969 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/969/)
          MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. (Revision 1294436)

          Result = UNSTABLE
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294436
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #969 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/969/ ) MAPREDUCE-2855 . Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. (Revision 1294436) Result = UNSTABLE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294436 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #182 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/182/)
          MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth.
          svn merge --ignore-ancestry -c 1294436 ../../trunk (Revision 1294437)

          Result = SUCCESS
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294437
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #182 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/182/ ) MAPREDUCE-2855 . Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. svn merge --ignore-ancestry -c 1294436 ../../trunk (Revision 1294437) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294437 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #606 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/606/)
          MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth.
          svn merge --ignore-ancestry -c 1294436 ../../trunk (Revision 1294437)

          Result = ABORTED
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294437
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #606 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/606/ ) MAPREDUCE-2855 . Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. svn merge --ignore-ancestry -c 1294436 ../../trunk (Revision 1294437) Result = ABORTED vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294437 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1801 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1801/)
          MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. (Revision 1294436)

          Result = ABORTED
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294436
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1801 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1801/ ) MAPREDUCE-2855 . Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. (Revision 1294436) Result = ABORTED vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294436 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Release Note Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time.
          Fix Version/s 0.23.2 [ 12319851 ]
          Resolution Fixed [ 1 ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Just committed to trunk, branch-0.23 and branch-0.23.2. Thanks Sid for the patch and the validation!

          Thanks Todd for filing the ticket and suggesting a fix!

          Show
          Vinod Kumar Vavilapalli added a comment - Just committed to trunk, branch-0.23 and branch-0.23.2. Thanks Sid for the patch and the validation! Thanks Todd for filing the ticket and suggesting a fix!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #592 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/592/)
          MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth.
          svn merge --ignore-ancestry -c 1294436 ../../trunk (Revision 1294437)

          Result = SUCCESS
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294437
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #592 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/592/ ) MAPREDUCE-2855 . Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. svn merge --ignore-ancestry -c 1294436 ../../trunk (Revision 1294437) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294437 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1865 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1865/)
          MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. (Revision 1294436)

          Result = SUCCESS
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294436
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1865 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1865/ ) MAPREDUCE-2855 . Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. (Revision 1294436) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294436 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1791 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1791/)
          MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. (Revision 1294436)

          Result = SUCCESS
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294436
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1791 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1791/ ) MAPREDUCE-2855 . Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. (Revision 1294436) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294436 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #605 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/605/)
          MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth.
          svn merge --ignore-ancestry -c 1294436 ../../trunk (Revision 1294437)

          Result = SUCCESS
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294437
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #605 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/605/ ) MAPREDUCE-2855 . Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. svn merge --ignore-ancestry -c 1294436 ../../trunk (Revision 1294437) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294437 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Nothing much to review here smile

          Looks good, +1.

          On another note, do we really need to use ResourceBundles for the counter names, or is this something we can do away with?

          Great question. Not even sure if anybody uses this at all. I guess we should just file a ticket and see if someone responds.

          Pushing this in.

          Show
          Vinod Kumar Vavilapalli added a comment - Nothing much to review here smile Looks good, +1. On another note, do we really need to use ResourceBundles for the counter names, or is this something we can do away with? Great question. Not even sure if anybody uses this at all. I guess we should just file a ticket and see if someone responds. Pushing this in.
          Vinod Kumar Vavilapalli made changes -
          Assignee Siddharth Seth [ sseth ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Looking at the patch for review/commit now..

          Show
          Vinod Kumar Vavilapalli added a comment - Looking at the patch for review/commit now..
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12515689/MR2855.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1913//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1913//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515689/MR2855.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1913//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1913//console This message is automatically generated.
          Siddharth Seth made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.23.2 [ 12319851 ]
          Labels mrv2
          Siddharth Seth made changes -
          Attachment MR2855.txt [ 12515689 ]
          Hide
          Siddharth Seth added a comment -

          Nice find Todd. These calls were taking up a good amount of time.
          Attaching a patch which passes the context classloader.

          For a 70MB, 11700 task history file.
          ParseTime: ~4.5 seconds (remains unchanged)
          Creating all Task objects: Comes down from ~11.3 seconds to ~4 seconds.
          MR-3901 does defer these calls to when they're actually required.

          On another note, do we really need to use ResourceBundles for the counter names, or is this something we can do away with?

          Show
          Siddharth Seth added a comment - Nice find Todd. These calls were taking up a good amount of time. Attaching a patch which passes the context classloader. For a 70MB, 11700 task history file. ParseTime: ~4.5 seconds (remains unchanged) Creating all Task objects: Comes down from ~11.3 seconds to ~4 seconds. MR-3901 does defer these calls to when they're actually required. On another note, do we really need to use ResourceBundles for the counter names, or is this something we can do away with?
          Todd Lipcon made changes -
          Field Original Value New Value
          Assignee Todd Lipcon [ tlipcon ]
          Hide
          Todd Lipcon added a comment -

          This is taking the majority of time when loading job history for large jobs. I see the following stack:

            Stack:
              java.util.ResourceBundle.getClassContext(Native Method)
              java.util.ResourceBundle.getLoader(ResourceBundle.java:400)
              java.util.ResourceBundle.getBundle(ResourceBundle.java:705)
              org.apache.hadoop.mapreduce.util.ResourceBundles.getBundle(ResourceBundles.java:36)
              org.apache.hadoop.mapreduce.util.ResourceBundles.getValue(ResourceBundles.java:54)
              org.apache.hadoop.mapreduce.util.ResourceBundles.getCounterName(ResourceBundles.java:87)
              org.apache.hadoop.mapreduce.counters.FileSystemCounterGroup$FSCounter.localizeCounterName(FileSystemCounterGroup.java:85)
              org.apache.hadoop.mapreduce.counters.FileSystemCounterGroup$FSCounter.getDisplayName(FileSystemCounterGroup.java:81)
              org.apache.hadoop.mapreduce.TypeConverter.toYarn(TypeConverter.java:275)
              org.apache.hadoop.mapreduce.v2.hs.CompletedTask.<init>(CompletedTask.java:89)
              org.apache.hadoop.mapreduce.v2.hs.CompletedJob.loadFullHistoryData(CompletedJob.java:270)
              org.apache.hadoop.mapreduce.v2.hs.CompletedJob.<init>(CompletedJob.java:90)
              org.apache.hadoop.mapreduce.v2.hs.JobHistory.loadJob(JobHistory.java:657)
              org.apache.hadoop.mapreduce.v2.hs.JobHistory.findJob(JobHistory.java:737)
              org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:903)
              org.apache.hadoop.mapreduce.v2.app.webapp.AppController.requireJob(AppController.java:313)
          
          Show
          Todd Lipcon added a comment - This is taking the majority of time when loading job history for large jobs. I see the following stack: Stack: java.util.ResourceBundle.getClassContext(Native Method) java.util.ResourceBundle.getLoader(ResourceBundle.java:400) java.util.ResourceBundle.getBundle(ResourceBundle.java:705) org.apache.hadoop.mapreduce.util.ResourceBundles.getBundle(ResourceBundles.java:36) org.apache.hadoop.mapreduce.util.ResourceBundles.getValue(ResourceBundles.java:54) org.apache.hadoop.mapreduce.util.ResourceBundles.getCounterName(ResourceBundles.java:87) org.apache.hadoop.mapreduce.counters.FileSystemCounterGroup$FSCounter.localizeCounterName(FileSystemCounterGroup.java:85) org.apache.hadoop.mapreduce.counters.FileSystemCounterGroup$FSCounter.getDisplayName(FileSystemCounterGroup.java:81) org.apache.hadoop.mapreduce.TypeConverter.toYarn(TypeConverter.java:275) org.apache.hadoop.mapreduce.v2.hs.CompletedTask.<init>(CompletedTask.java:89) org.apache.hadoop.mapreduce.v2.hs.CompletedJob.loadFullHistoryData(CompletedJob.java:270) org.apache.hadoop.mapreduce.v2.hs.CompletedJob.<init>(CompletedJob.java:90) org.apache.hadoop.mapreduce.v2.hs.JobHistory.loadJob(JobHistory.java:657) org.apache.hadoop.mapreduce.v2.hs.JobHistory.findJob(JobHistory.java:737) org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:903) org.apache.hadoop.mapreduce.v2.app.webapp.AppController.requireJob(AppController.java:313)
          Todd Lipcon created issue -

            People

            • Assignee:
              Siddharth Seth
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development