Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1635

ResourceEstimator does not work after MAPREDUCE-842

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: tasktracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed a bug related to resource estimation for disk-based scheduling by modifying TaskTracker to return correct map output size for the completed maps and -1 for other tasks or failures.

      Description

      MAPREDUCE-842 changed Child's mapred.local.dir to have attemptDir as the base local directory. Also assumption is that
      org.apache.hadoop.mapred.MapOutputFile always gets Child's mapred.local.dir.
      But, MapOuptutFile.getOutputFile() is called from TaskTracker's conf, which does not find the output file. Thus TaskTracker.tryToGetOutputSize() always returns -1.

      1. patch-1635-1.txt
        9 kB
        Amareshwari Sriramadasu
      2. patch-1635.txt
        8 kB
        Amareshwari Sriramadasu
      3. ASF.LICENSE.NOT.GRANTED--patch-1635-ydist.txt
        9 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          Amareshwari Sriramadasu created issue -
          Amareshwari Sriramadasu made changes -
          Field Original Value New Value
          Attachment patch-1635.txt [ 12440046 ]
          Amareshwari Sriramadasu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Amareshwari Sriramadasu [ amareshwari ]
          Amareshwari Sriramadasu made changes -
          Link This issue incorporates MAPREDUCE-1321 [ MAPREDUCE-1321 ]
          Amareshwari Sriramadasu made changes -
          Description MAPREDUCE-842 changed Child's mapred.local.dir to have attemptDir as the base local directory. Also assumption is that
          org.apache.hadoop.mapred.MapOutputFile always gets Child's mapred.local.dir.
          But, MapOuptutFile.getOutputFile() is called from TaskTracker's conf, which does not find the output file. Thus TaskTracker.tryToGetOutputSize() always returns zero.
          MAPREDUCE-842 changed Child's mapred.local.dir to have attemptDir as the base local directory. Also assumption is that
          org.apache.hadoop.mapred.MapOutputFile always gets Child's mapred.local.dir.
          But, MapOuptutFile.getOutputFile() is called from TaskTracker's conf, which does not find the output file. Thus TaskTracker.tryToGetOutputSize() always returns -1.
          Amareshwari Sriramadasu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Amareshwari Sriramadasu made changes -
          Attachment patch-1635-1.txt [ 12440447 ]
          Amareshwari Sriramadasu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue relates to MAPREDUCE-1662 [ MAPREDUCE-1662 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Vinod Kumar Vavilapalli made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags [Reviewed]
          Amareshwari Sriramadasu made changes -
          Attachment patch-1635-ydist.txt [ 12441448 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Amareshwari Sriramadasu made changes -
          Release Note Fixed a bug in TaskTracker to return correct map output size for the completed maps and -1 for other tasks or failures.
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12314045 ]
          Fix Version/s 0.22.0 [ 12314184 ]
          Vinod Kumar Vavilapalli made changes -
          Release Note Fixed a bug in TaskTracker to return correct map output size for the completed maps and -1 for other tasks or failures. Fixed a bug related to resource estimation for disk-based scheduling by modifying TaskTracker to return correct map output size for the completed maps and -1 for other tasks or failures.
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              Amareshwari Sriramadasu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development