Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-543

large pending jobs hog resources

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • None
    • 0.21.0
    • None
    • None

    Description

      observing the cluster over the last day - one thing i noticed is that small jobs (single digit tasks) are not doing a good job competing against large jobs. what seems to happen is that:

      • large job comes along and needs to wait for a while for other large jobs.
      • slots are slowly transfered from one large job to another
      • small tasks keep waiting forever.

      is this an artifact of deficit based scheduling? it seems that long pending large jobs are out-scheduling small jobs

      Attachments

        Activity

          People

            matei Matei Alexandru Zaharia
            jsensarma Joydeep Sen Sarma
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: