Hadoop Common
  1. Hadoop Common
  2. HADOOP-6439

Shuffle deadlocks on wrong number of maps

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: conf
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:

      09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
      09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps

      But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

      1. HADOOP-6439-7.patch
        21 kB
        Hemanth Yamijala
      2. ASF.LICENSE.NOT.GRANTED--HADOOP-6439-6.patch
        21 kB
        V.V.Chaitanya Krishna
      3. HADOOP-6439-6.patch
        21 kB
        V.V.Chaitanya Krishna
      4. HADOOP-6439-5.patch
        21 kB
        V.V.Chaitanya Krishna
      5. HADOOP-6439-4.patch
        20 kB
        V.V.Chaitanya Krishna
      6. HADOOP-6439-3.patch
        20 kB
        V.V.Chaitanya Krishna
      7. HADOOP-6439-2.patch
        19 kB
        V.V.Chaitanya Krishna
      8. HADOOP-6439-1.patch
        17 kB
        V.V.Chaitanya Krishna
      9. mr-1252.patch
        0.8 kB
        Owen O'Malley

        Issue Links

          Activity

          Owen O'Malley created issue -
          Owen O'Malley made changes -
          Field Original Value New Value
          Attachment mr-1252.patch [ 12426479 ]
          Owen O'Malley made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Owen O'Malley made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ravi Gummadi made changes -
          Project Hadoop Map/Reduce [ 12310941 ] Hadoop Common [ 12310240 ]
          Key MAPREDUCE-1252 HADOOP-6439
          Affects Version/s 0.21.0 [ 12313563 ]
          Affects Version/s 0.22.0 [ 12314296 ]
          Affects Version/s 0.21.0 [ 12314045 ]
          Affects Version/s 0.22.0 [ 12314184 ]
          Fix Version/s 0.21.0 [ 12313563 ]
          Fix Version/s 0.22.0 [ 12314296 ]
          Fix Version/s 0.21.0 [ 12314045 ]
          Fix Version/s 0.22.0 [ 12314184 ]
          Component/s conf [ 12310711 ]
          Component/s task [ 12312920 ]
          Ravi Gummadi made changes -
          Link This issue is related to HADOOP-6105 [ HADOOP-6105 ]
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6439-1.patch [ 12428265 ]
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6439-2.patch [ 12428273 ]
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6439-3.patch [ 12428406 ]
          Amareshwari Sriramadasu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Owen O'Malley [ owen.omalley ] V.V.Chaitanya Krishna [ chaitk ]
          Hemanth Yamijala made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6439-4.patch [ 12429328 ]
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6439-5.patch [ 12429519 ]
          V.V.Chaitanya Krishna made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6439-6.patch [ 12429634 ]
          V.V.Chaitanya Krishna made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          V.V.Chaitanya Krishna made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          V.V.Chaitanya Krishna made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6439-6.patch [ 12441685 ]
          V.V.Chaitanya Krishna made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          V.V.Chaitanya Krishna made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          V.V.Chaitanya Krishna made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hemanth Yamijala made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hemanth Yamijala made changes -
          Attachment HADOOP-6439-7.patch [ 12442271 ]
          Hemanth Yamijala made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hemanth Yamijala made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.21.0 [ 12313563 ]
          Resolution Fixed [ 1 ]
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12313563 ]
          Fix Version/s 0.22.0 [ 12314296 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              V.V.Chaitanya Krishna
              Reporter:
              Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development