Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-177

Hadoop performance degrades significantly as more and more jobs complete



    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • None
    • None
    • None
    • None


      When I ran the gridmix 2 benchmark load on a fresh cluster of 500 nodes with hadoop trunk,
      the gridmix load, consisting of 202 map/reduce jobs of various sizes, completed in 32 minutes.
      Then I ran the same set of the jobs on the same cluster, yhey completed in 43 minutes.
      When I ran them the third times, it took (almost) forever — the job tracker became non-responsive.

      The job tracker's heap size was set to 2GB.
      The cluster is configured to keep up to 500 jobs in memory.

      The job tracker kept one cpu busy all the time. Look like it was due to GC.

      I believe the release 0.18/0.19 have the similar behavior.

      I believe 0.18 and 0.18 also have the similar behavior.


        1. map_scheduling_rate.txt
          4 kB
          Runping Qi
        2. HADOOP-4766-v3.4-0.19.patch
          27 kB
          Amar Kamat
        3. HADOOP-4766-v2.8-0.19.patch
          24 kB
          Amar Kamat
        4. HADOOP-4766-v2.8-0.18.patch
          19 kB
          Amar Kamat
        5. HADOOP-4766-v2.8.patch
          23 kB
          Amar Kamat
        6. HADOOP-4766-v2.7-0.19.patch
          22 kB
          Amar Kamat
        7. HADOOP-4766-v2.7-0.18.patch
          16 kB
          Amar Kamat
        8. HADOOP-4766-v2.7.patch
          21 kB
          Amar Kamat
        9. HADOOP-4766-v2.6.patch
          21 kB
          Amar Kamat
        10. HADOOP-4766-v2.4.patch
          21 kB
          Amar Kamat
        11. HADOOP-4766-v2.10.patch
          22 kB
          Amar Kamat
        12. HADOOP-4766-v1.patch
          3 kB
          Amar Kamat

        Issue Links



              ikoltsidas Ioannis Koltsidas
              runping Runping Qi
              3 Vote for this issue
              28 Start watching this issue