MRUnit
  1. MRUnit
  2. MRUNIT-165

MapReduceDriver calls Mapper#cleanup for each input instead of once

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 1.0.0
    • Labels:
      None

      Description

      MapReduceDriver calls the Mapper#run method for each input, causing the Mapper#cleanup method to be called multiple times.

      I believe this is a bug, since the contract in MapReduce is that, for a single Mapper instance, the Mapper#cleanup method is only called once after all inputs to that mapper have been processed. I might be mistaken in my assumption here.

      This would not be an issue, were it not for the fact that MapReduceDriver has only a single instance of Mapper.

      One solution might be to pass the Mapper class into the MapReduceDriver and create a new instance for each input. Another solution might be to call the MapDriver with multiple inputs (which AFAIK is not possible).

      See attached patch for an example of a stateful mapper and a test which fails due to the bug.

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in mrunit-trunk #506 (See https://builds.apache.org/job/mrunit-trunk/506/)
          MRUNIT-165: MapReduceDriver calls Mapper#cleanup for each input instead of once (unit test contributed by Yoni Ben-Meshulam) (Revision 92ad229dafea47db18fa978e809d13261721a806)

          Result = SUCCESS
          dbeech : https://git-wip-us.apache.org/repos/asf?p=mrunit.git&a=commit&h=92ad229dafea47db18fa978e809d13261721a806
          Files :

          • src/test/java/org/apache/hadoop/mrunit/mapreduce/StatefulMapper.java
          • src/test/java/org/apache/hadoop/mrunit/mapreduce/TestStatefulMapReduce.java
          Show
          Hudson added a comment - Integrated in mrunit-trunk #506 (See https://builds.apache.org/job/mrunit-trunk/506/ ) MRUNIT-165 : MapReduceDriver calls Mapper#cleanup for each input instead of once (unit test contributed by Yoni Ben-Meshulam) (Revision 92ad229dafea47db18fa978e809d13261721a806) Result = SUCCESS dbeech : https://git-wip-us.apache.org/repos/asf?p=mrunit.git&a=commit&h=92ad229dafea47db18fa978e809d13261721a806 Files : src/test/java/org/apache/hadoop/mrunit/mapreduce/StatefulMapper.java src/test/java/org/apache/hadoop/mrunit/mapreduce/TestStatefulMapReduce.java
          Hide
          Yoni Ben-Meshulam added a comment -

          Awesome. Thanks Dave.

          Show
          Yoni Ben-Meshulam added a comment - Awesome. Thanks Dave.
          Hide
          Dave Beech added a comment -

          Thanks for the patch. I've confirmed the test does fail against the 0.9.0 code but it passes on trunk. The recent multiple inputs changes in MRUNIT-64 seem to have fixed this bug as a side effect.

          Show
          Dave Beech added a comment - Thanks for the patch. I've confirmed the test does fail against the 0.9.0 code but it passes on trunk. The recent multiple inputs changes in MRUNIT-64 seem to have fixed this bug as a side effect.
          Hide
          Yoni Ben-Meshulam added a comment -

          Attaching a simple stateful mapper and a test which fails due to the bug in MapReduceDriver.

          Show
          Yoni Ben-Meshulam added a comment - Attaching a simple stateful mapper and a test which fails due to the bug in MapReduceDriver.
          Hide
          Dave Beech added a comment -

          Hi Yoni - could you please post your unit test code that exhibits this bug? I think this may not be a problem any longer since we now support multiple inputs, but I'd like to run your test again using the latest code to make sure. Thanks!

          Show
          Dave Beech added a comment - Hi Yoni - could you please post your unit test code that exhibits this bug? I think this may not be a problem any longer since we now support multiple inputs, but I'd like to run your test again using the latest code to make sure. Thanks!

            People

            • Assignee:
              Dave Beech
              Reporter:
              Yoni Ben-Meshulam
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development