Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6473

Job submission can take a long time during Cluster initialization

VotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • performance
    • None
    • Reviewed

    Description

      During initialization in Cluster.java, the framework provider classes are loaded inside a sync block which can considerably increase job submission time when the number of submissions are high. The motive is to reduce time spent in this sync block safely to improve performance.

      synchronized (frameworkLoader) {
            for (ClientProtocolProvider provider : frameworkLoader) {
              LOG.debug("Trying ClientProtocolProvider : "
                  + provider.getClass().getName());
      

      Attachments

        1. 99%ile.png
          9 kB
          Kuhu Shukla
        2. avgtime.png
          9 kB
          Kuhu Shukla
        3. MAPREDUCE-6473-PerfTest.txt
          4 kB
          Kuhu Shukla
        4. MAPREDUCE-6473-v1.patch
          2 kB
          Kuhu Shukla
        5. MAPREDUCE-6473-v2.patch
          3 kB
          Kuhu Shukla
        6. MAPREDUCE-6473-v3.patch
          3 kB
          Kuhu Shukla
        7. MAPREDUCE-6473-v4.patch
          3 kB
          Kuhu Shukla
        8. MAPREDUCE-6473-v5.patch
          4 kB
          Kuhu Shukla
        9. MAPREDUCE-6473-v6.patch
          4 kB
          Kuhu Shukla
        10. MAPREDUCE-6473-v7.patch
          4 kB
          Kuhu Shukla

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            kshukla Kuhu Shukla
            kshukla Kuhu Shukla
            Votes:
            0 Vote for this issue
            Watchers:
            6 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment