Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6473

Job submission can take a long time during Cluster initialization

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: performance
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      During initialization in Cluster.java, the framework provider classes are loaded inside a sync block which can considerably increase job submission time when the number of submissions are high. The motive is to reduce time spent in this sync block safely to improve performance.

      synchronized (frameworkLoader) {
            for (ClientProtocolProvider provider : frameworkLoader) {
              LOG.debug("Trying ClientProtocolProvider : "
                  + provider.getClass().getName());
      

        Attachments

        1. 99%ile.png
          9 kB
          Kuhu Shukla
        2. avgtime.png
          9 kB
          Kuhu Shukla
        3. MAPREDUCE-6473-PerfTest.txt
          4 kB
          Kuhu Shukla
        4. MAPREDUCE-6473-v1.patch
          2 kB
          Kuhu Shukla
        5. MAPREDUCE-6473-v2.patch
          3 kB
          Kuhu Shukla
        6. MAPREDUCE-6473-v3.patch
          3 kB
          Kuhu Shukla
        7. MAPREDUCE-6473-v4.patch
          3 kB
          Kuhu Shukla
        8. MAPREDUCE-6473-v5.patch
          4 kB
          Kuhu Shukla
        9. MAPREDUCE-6473-v6.patch
          4 kB
          Kuhu Shukla
        10. MAPREDUCE-6473-v7.patch
          4 kB
          Kuhu Shukla

          Issue Links

            Activity

              People

              • Assignee:
                kshukla Kuhu Shukla
                Reporter:
                kshukla Kuhu Shukla
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: