Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-6439

Shuffle deadlocks on wrong number of maps

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: conf
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:

      09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
      09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps

      But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

        Attachments

        1. mr-1252.patch
          0.8 kB
          Owen O'Malley
        2. HADOOP-6439-1.patch
          17 kB
          V.V.Chaitanya Krishna
        3. HADOOP-6439-2.patch
          19 kB
          V.V.Chaitanya Krishna
        4. HADOOP-6439-3.patch
          20 kB
          V.V.Chaitanya Krishna
        5. HADOOP-6439-4.patch
          20 kB
          V.V.Chaitanya Krishna
        6. HADOOP-6439-5.patch
          21 kB
          V.V.Chaitanya Krishna
        7. HADOOP-6439-6.patch
          21 kB
          V.V.Chaitanya Krishna
        8. ASF.LICENSE.NOT.GRANTED--HADOOP-6439-6.patch
          21 kB
          V.V.Chaitanya Krishna
        9. HADOOP-6439-7.patch
          21 kB
          Hemanth Yamijala

          Issue Links

            Activity

              People

              • Assignee:
                chaitk V.V.Chaitanya Krishna
                Reporter:
                owen.omalley Owen O'Malley
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: