Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1073

Progress reported for pipes tasks is incorrect.

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.1
    • Fix Version/s: None
    • Component/s: pipes
    • Labels:
      None

      Description

      Currently in pipes, org.apache.hadoop.mapred.pipes.PipesMapRunner.run(RecordReader<K1, V1>, OutputCollector<K2, V2>, Reporter) we do the following:

              while (input.next(key, value)) {
                downlink.mapItem(key, value);
                if(skipping) {
                  downlink.flush();
                }
              }
      

      This would result in consumption of all the records for current task and taking task progress to 100% whereas the actual pipes application would be trailing behind.

      1. MAPREDUCE-1073_yhadoop20.patch
        1 kB
        Arun C Murthy
      2. mapreduce-1073--2010-03-31.patch
        2 kB
        Dick King
      3. mapreduce-1073--2010-04-06.patch
        282 kB
        Dick King
      4. MAPREDUCE-1073--yhadoop20--2010-07-22.patch
        27 kB
        Dick King
      5. MAPREDUCE-1073--yhadoop20--2010-07-22--1530.patch
        26 kB
        Dick King

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Dick King
            Reporter:
            Sreekanth Ramakrishnan
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:

              Development