[MAPREDUCE-6473] Job submission can take a long time during Cluster initialization

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: performance
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Kuhu Shukla
• Reporter: Kuhu Shukla
• Created: 09/Sep/15 21:49
• Updated: 30/Aug/16 01:16
• Resolved: 13/Jan/16 00:00

Description

During initialization in Cluster.java, the framework provider classes are loaded inside a sync block which can considerably increase job submission time when the number of submissions are high. The motive is to reduce time spent in this sync block safely to improve performance.

synchronized (frameworkLoader) {
      for (ClientProtocolProvider provider : frameworkLoader) {
        LOG.debug("Trying ClientProtocolProvider : "
            + provider.getClass().getName());

Issue Links

• relates to: MAPREDUCE-6761 Regression when handling providers - invalid configuration ServiceConfiguration causes Cluster initialization failure