Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-6439

Shuffle deadlocks on wrong number of maps

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.21.0, 0.22.0
    • 0.21.0
    • conf
    • None
    • Reviewed

    Description

      The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:

      09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
      09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps

      But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

      Attachments

        1. ASF.LICENSE.NOT.GRANTED--HADOOP-6439-6.patch
          21 kB
          V.V.Chaitanya Krishna
        2. HADOOP-6439-1.patch
          17 kB
          V.V.Chaitanya Krishna
        3. HADOOP-6439-2.patch
          19 kB
          V.V.Chaitanya Krishna
        4. HADOOP-6439-3.patch
          20 kB
          V.V.Chaitanya Krishna
        5. HADOOP-6439-4.patch
          20 kB
          V.V.Chaitanya Krishna
        6. HADOOP-6439-5.patch
          21 kB
          V.V.Chaitanya Krishna
        7. HADOOP-6439-6.patch
          21 kB
          V.V.Chaitanya Krishna
        8. HADOOP-6439-7.patch
          21 kB
          Hemanth Yamijala
        9. mr-1252.patch
          0.8 kB
          Owen O'Malley

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            chaitk V.V.Chaitanya Krishna
            omalley Owen O'Malley
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment