Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4565

Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.3
    • Fix Version/s: 1.2.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time.

      Description

      Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.

      1. branch1-MR-4565.patch
        3 kB
        Karthik Kambatla
      2. branch1-MR-4565.patch
        3 kB
        Karthik Kambatla

        Issue Links

          Activity

          Hide
          kkambatl Karthik Kambatla (Inactive) added a comment -

          Uploading a patch very similar to the one posted for MR-2855.

          Show
          kkambatl Karthik Kambatla (Inactive) added a comment - Uploading a patch very similar to the one posted for MR-2855.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541499/branch1-MR-4565.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2749//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541499/branch1-MR-4565.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2749//console This message is automatically generated.
          Hide
          sseth Siddharth Seth added a comment -

          Looks good, except for one minor nit. Since CounterGroup is a public class, getResourceBundle should be annotated as @Private, or leave things as is and update the functions in both classes.

          Show
          sseth Siddharth Seth added a comment - Looks good, except for one minor nit. Since CounterGroup is a public class, getResourceBundle should be annotated as @Private, or leave things as is and update the functions in both classes.
          Hide
          kkambatl Karthik Kambatla (Inactive) added a comment -

          Thanks for reviewing it, Sid. My bad to have missed the annotation.

          Uploading a patch with the annotation.

          Show
          kkambatl Karthik Kambatla (Inactive) added a comment - Thanks for reviewing it, Sid. My bad to have missed the annotation. Uploading a patch with the annotation.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541823/branch1-MR-4565.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2754//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541823/branch1-MR-4565.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2754//console This message is automatically generated.
          Hide
          sseth Siddharth Seth added a comment -

          +1

          Show
          sseth Siddharth Seth added a comment - +1
          Hide
          sseth Siddharth Seth added a comment -

          Committed to branch-1. Thanks Karthik!

          Show
          sseth Siddharth Seth added a comment - Committed to branch-1. Thanks Karthik!
          Hide
          kkambatl Karthik Kambatla (Inactive) added a comment -

          Sid, thanks for taking care of this.

          Show
          kkambatl Karthik Kambatla (Inactive) added a comment - Sid, thanks for taking care of this.
          Hide
          mattf Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          mattf Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.

            People

            • Assignee:
              kasha Karthik Kambatla
              Reporter:
              kasha Karthik Kambatla
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development