Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4565

Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.3
    • Fix Version/s: 1.2.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time.

      Description

      Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.

      1. branch1-MR-4565.patch
        3 kB
        Karthik Kambatla
      2. branch1-MR-4565.patch
        3 kB
        Karthik Kambatla

        Issue Links

          Activity

          Karthik Kambatla (Inactive) created issue -
          Karthik Kambatla (Inactive) made changes -
          Field Original Value New Value
          Link This issue is a clone of MAPREDUCE-2855 [ MAPREDUCE-2855 ]
          Karthik Kambatla (Inactive) made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Issue Type Bug [ 1 ] Improvement [ 4 ]
          Labels mrv2
          Fix Version/s 0.23.2 [ 12319851 ]
          Affects Version/s 1.0.3 [ 12320250 ]
          Affects Version/s 0.23.0 [ 12315570 ]
          Target Version/s 0.23.2 [ 12319851 ]
          Karthik Kambatla (Inactive) made changes -
          Assignee Siddharth Seth [ sseth ] Karthik Kambatla [ kkambatl ]
          Karthik Kambatla (Inactive) made changes -
          Attachment branch1-MR-4565.patch [ 12541499 ]
          Karthik Kambatla (Inactive) made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Karthik Kambatla (Inactive) made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Karthik Kambatla (Inactive) made changes -
          Attachment branch1-MR-4565.patch [ 12541823 ]
          Siddharth Seth made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 1.2.0 [ 12321661 ]
          Resolution Fixed [ 1 ]
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Gavin made changes -
          Assignee Karthik Kambatla [ kkambatl ] Karthik Kambatla [ kasha ]
          Gavin made changes -
          Reporter Karthik Kambatla [ kkambatl ] Karthik Kambatla [ kasha ]

            People

            • Assignee:
              Karthik Kambatla
              Reporter:
              Karthik Kambatla
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development