Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6473

Job submission can take a long time during Cluster initialization

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: performance
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      During initialization in Cluster.java, the framework provider classes are loaded inside a sync block which can considerably increase job submission time when the number of submissions are high. The motive is to reduce time spent in this sync block safely to improve performance.

      synchronized (frameworkLoader) {
            for (ClientProtocolProvider provider : frameworkLoader) {
              LOG.debug("Trying ClientProtocolProvider : "
                  + provider.getClass().getName());
      

        Attachments

        1. 99%ile.png
          9 kB
          Kuhu Shukla
        2. avgtime.png
          9 kB
          Kuhu Shukla
        3. MAPREDUCE-6473-PerfTest.txt
          4 kB
          Kuhu Shukla
        4. MAPREDUCE-6473-v1.patch
          2 kB
          Kuhu Shukla
        5. MAPREDUCE-6473-v2.patch
          3 kB
          Kuhu Shukla
        6. MAPREDUCE-6473-v3.patch
          3 kB
          Kuhu Shukla
        7. MAPREDUCE-6473-v4.patch
          3 kB
          Kuhu Shukla
        8. MAPREDUCE-6473-v5.patch
          4 kB
          Kuhu Shukla
        9. MAPREDUCE-6473-v6.patch
          4 kB
          Kuhu Shukla
        10. MAPREDUCE-6473-v7.patch
          4 kB
          Kuhu Shukla

        Issue Links

          Activity

            People

            • Assignee:
              kshukla Kuhu Shukla
              Reporter:
              kshukla Kuhu Shukla

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment