Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2002

MRUnit driver classes should provide ability to set a configuration object to be passed into the mapper/reducer

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.20.2
    • Fix Version/s: None
    • Component/s: contrib/mrunit
    • Labels:
      None

      Description

      Short description:

      Enhance the org.apache.hadoop.mrunit.mapreduce.MapDriver, ReduceDriver, and MapReduceDriver unit test driver classes to contain "setConfiguration" and "withConfiguration" methods for passing in user-supplied org.apache.hadoop.conf.Configuration objects, and have those configuration objects eventually get passed on to the Context objects that are passed in to the mapper/reducer "setup" methods. (Rather than passing in an empty Configuration object, as is being done now.)

      Long description:

      The MRUnit driver classes (i.e., MapDriver, ReduceDriver, and MapReduceDriver) ought to be enhanced to contain methods for setting a Configuration object to be used by the mapper/reducer being tested - i.e., setConfiguration() and withConfiguration().

      The only way to effectively pass parameters into a mapper or reducer is by setting properties on a configuration object, which the mapper/reducer can then retrieve in their "setup" step, and use to customize its operation. As a result, specific mappers/reducers may require the presence of specific configuration properties/parameters in order to function correctly (or at all). (I am currently coding such a reducer right now.)

      Testing such a mapper/reducer thus requires that the unit testing framework used provide the ability to pass in user-supplied Configuration objects to them so that they can be tested with appropriate parameter values. However, MRUnit currently does not provide this ability. (All mappers/reducers are always passed an empty configuration object.) And there is not even currently any (easy) way for the end-user to fix this problem by creating a simple sub-class that supplies this functionality, as such subclasses would require a substantial reimplementation/override of several MRUnit framework classes.

      I believe this something that is not too difficult to fix in the MRUnit framework code, however, and would greatly help the usability of MRUnit.

      Although I don't have time to code this enhancement right now, if needed/preferred I could squeeze out some time to code up a patch for this. If that's needed, please let me know.

        Issue Links

          Activity

          David Rosenstrauch created issue -
          Jim Donofrio made changes -
          Field Original Value New Value
          Link This issue duplicates MRUNIT-86 [ MRUNIT-86 ]
          Jim Donofrio made changes -
          Link This issue duplicates MAPREDUCE-1569 [ MAPREDUCE-1569 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              David Rosenstrauch
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:

                Development