Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-177

Hadoop performance degrades significantly as more and more jobs complete

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • None
    • None
    • None
    • None

    Description

      When I ran the gridmix 2 benchmark load on a fresh cluster of 500 nodes with hadoop trunk,
      the gridmix load, consisting of 202 map/reduce jobs of various sizes, completed in 32 minutes.
      Then I ran the same set of the jobs on the same cluster, yhey completed in 43 minutes.
      When I ran them the third times, it took (almost) forever — the job tracker became non-responsive.

      The job tracker's heap size was set to 2GB.
      The cluster is configured to keep up to 500 jobs in memory.

      The job tracker kept one cpu busy all the time. Look like it was due to GC.

      I believe the release 0.18/0.19 have the similar behavior.

      I believe 0.18 and 0.18 also have the similar behavior.

      Attachments

        1. map_scheduling_rate.txt
          4 kB
          Runping Qi
        2. HADOOP-4766-v1.patch
          3 kB
          Amar Kamat
        3. HADOOP-4766-v2.4.patch
          21 kB
          Amar Kamat
        4. HADOOP-4766-v2.6.patch
          21 kB
          Amar Kamat
        5. HADOOP-4766-v2.7.patch
          21 kB
          Amar Kamat
        6. HADOOP-4766-v2.7-0.19.patch
          22 kB
          Amar Kamat
        7. HADOOP-4766-v2.7-0.18.patch
          16 kB
          Amar Kamat
        8. HADOOP-4766-v2.8.patch
          23 kB
          Amar Kamat
        9. HADOOP-4766-v2.8-0.19.patch
          24 kB
          Amar Kamat
        10. HADOOP-4766-v2.8-0.18.patch
          19 kB
          Amar Kamat
        11. HADOOP-4766-v2.10.patch
          22 kB
          Amar Kamat
        12. HADOOP-4766-v3.4-0.19.patch
          27 kB
          Amar Kamat

        Issue Links

          Activity

            People

              ikoltsidas Ioannis Koltsidas
              runping Runping Qi
              Votes:
              3 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: