Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4565

Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.3
    • Fix Version/s: 1.2.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time.

      Description

      Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.

      1. branch1-MR-4565.patch
        3 kB
        Karthik Kambatla
      2. branch1-MR-4565.patch
        3 kB
        Karthik Kambatla

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open In Progress In Progress
          6m 29s 1 Karthik Kambatla (Inactive) 18/Aug/12 19:29
          In Progress In Progress Patch Available Patch Available
          27s 1 Karthik Kambatla (Inactive) 18/Aug/12 19:29
          Patch Available Patch Available Resolved Resolved
          3d 3h 17m 1 Siddharth Seth 21/Aug/12 22:46
          Resolved Resolved Closed Closed
          266d 6h 29m 1 Matt Foley 15/May/13 05:15
          Gavin made changes -
          Reporter Karthik Kambatla [ kkambatl ] Karthik Kambatla [ kasha ]
          Gavin made changes -
          Assignee Karthik Kambatla [ kkambatl ] Karthik Kambatla [ kasha ]
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Sid, thanks for taking care of this.

          Show
          Karthik Kambatla (Inactive) added a comment - Sid, thanks for taking care of this.
          Siddharth Seth made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 1.2.0 [ 12321661 ]
          Resolution Fixed [ 1 ]
          Hide
          Siddharth Seth added a comment -

          Committed to branch-1. Thanks Karthik!

          Show
          Siddharth Seth added a comment - Committed to branch-1. Thanks Karthik!
          Hide
          Siddharth Seth added a comment -

          +1

          Show
          Siddharth Seth added a comment - +1
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541823/branch1-MR-4565.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2754//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541823/branch1-MR-4565.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2754//console This message is automatically generated.
          Karthik Kambatla (Inactive) made changes -
          Attachment branch1-MR-4565.patch [ 12541823 ]
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Thanks for reviewing it, Sid. My bad to have missed the annotation.

          Uploading a patch with the annotation.

          Show
          Karthik Kambatla (Inactive) added a comment - Thanks for reviewing it, Sid. My bad to have missed the annotation. Uploading a patch with the annotation.
          Hide
          Siddharth Seth added a comment -

          Looks good, except for one minor nit. Since CounterGroup is a public class, getResourceBundle should be annotated as @Private, or leave things as is and update the functions in both classes.

          Show
          Siddharth Seth added a comment - Looks good, except for one minor nit. Since CounterGroup is a public class, getResourceBundle should be annotated as @Private, or leave things as is and update the functions in both classes.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541499/branch1-MR-4565.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2749//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541499/branch1-MR-4565.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2749//console This message is automatically generated.
          Karthik Kambatla (Inactive) made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Karthik Kambatla (Inactive) made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Karthik Kambatla (Inactive) made changes -
          Attachment branch1-MR-4565.patch [ 12541499 ]
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Uploading a patch very similar to the one posted for MR-2855.

          Show
          Karthik Kambatla (Inactive) added a comment - Uploading a patch very similar to the one posted for MR-2855.
          Karthik Kambatla (Inactive) made changes -
          Assignee Siddharth Seth [ sseth ] Karthik Kambatla [ kkambatl ]
          Karthik Kambatla (Inactive) made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Issue Type Bug [ 1 ] Improvement [ 4 ]
          Labels mrv2
          Fix Version/s 0.23.2 [ 12319851 ]
          Affects Version/s 1.0.3 [ 12320250 ]
          Affects Version/s 0.23.0 [ 12315570 ]
          Target Version/s 0.23.2 [ 12319851 ]
          Karthik Kambatla (Inactive) made changes -
          Field Original Value New Value
          Link This issue is a clone of MAPREDUCE-2855 [ MAPREDUCE-2855 ]
          Karthik Kambatla (Inactive) created issue -

            People

            • Assignee:
              Karthik Kambatla
              Reporter:
              Karthik Kambatla
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development