Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4565

Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.3
    • Fix Version/s: 1.2.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time.

      Description

      Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.

      1. branch1-MR-4565.patch
        3 kB
        Karthik Kambatla
      2. branch1-MR-4565.patch
        3 kB
        Karthik Kambatla

        Issue Links

          Activity

          Hide
          Karthik Kambatla added a comment -

          Uploading a patch very similar to the one posted for MR-2855.

          Show
          Karthik Kambatla added a comment - Uploading a patch very similar to the one posted for MR-2855.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541499/branch1-MR-4565.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2749//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541499/branch1-MR-4565.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2749//console This message is automatically generated.
          Hide
          Siddharth Seth added a comment -

          Looks good, except for one minor nit. Since CounterGroup is a public class, getResourceBundle should be annotated as @Private, or leave things as is and update the functions in both classes.

          Show
          Siddharth Seth added a comment - Looks good, except for one minor nit. Since CounterGroup is a public class, getResourceBundle should be annotated as @Private, or leave things as is and update the functions in both classes.
          Hide
          Karthik Kambatla added a comment -

          Thanks for reviewing it, Sid. My bad to have missed the annotation.

          Uploading a patch with the annotation.

          Show
          Karthik Kambatla added a comment - Thanks for reviewing it, Sid. My bad to have missed the annotation. Uploading a patch with the annotation.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541823/branch1-MR-4565.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2754//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541823/branch1-MR-4565.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2754//console This message is automatically generated.
          Hide
          Siddharth Seth added a comment -

          +1

          Show
          Siddharth Seth added a comment - +1
          Hide
          Siddharth Seth added a comment -

          Committed to branch-1. Thanks Karthik!

          Show
          Siddharth Seth added a comment - Committed to branch-1. Thanks Karthik!
          Hide
          Karthik Kambatla added a comment -

          Sid, thanks for taking care of this.

          Show
          Karthik Kambatla added a comment - Sid, thanks for taking care of this.
          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.

            People

            • Assignee:
              Karthik Kambatla
              Reporter:
              Karthik Kambatla
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development