MRUnit
  1. MRUnit
  2. MRUNIT-14

Support JobConfigurable and Closable interfaces

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.8.0
    • Labels:
      None

      Description

      MRUnit should support the invocation of a mapper or reducer's configure() and close() methods as dictated by JobConfigurable and Closable respectively.

        Activity

        Hide
        Brock Noland added a comment -

        Committed in 1207646.

        Show
        Brock Noland added a comment - Committed in 1207646.
        Hide
        Jim Donofrio added a comment -

        I understand why you made this change for a JUnit test class with a single @Test method. However, calling configure and close prevents one from calling configure once in a @BeforeClass method and testing several (key, value) pairs in a series of separate @Test methods using the same static Mapper or Reduce instance. This more closely mirrors what the actual mapreduce does and allows one to make sure they are properly clearing state between different (key, value) pairs.

        This change also makes 0.8.x non backwards compatible with mapreduce jobs that test with mrunit 0.5.0 since you will call configure again with an empty Configuration object after we already called it.

        Maybe you could make calling configure and close configurable by us providing a boolean in a constructor to indicate whether you should call configure and close

        Show
        Jim Donofrio added a comment - I understand why you made this change for a JUnit test class with a single @Test method. However, calling configure and close prevents one from calling configure once in a @BeforeClass method and testing several (key, value) pairs in a series of separate @Test methods using the same static Mapper or Reduce instance. This more closely mirrors what the actual mapreduce does and allows one to make sure they are properly clearing state between different (key, value) pairs. This change also makes 0.8.x non backwards compatible with mapreduce jobs that test with mrunit 0.5.0 since you will call configure again with an empty Configuration object after we already called it. Maybe you could make calling configure and close configurable by us providing a boolean in a constructor to indicate whether you should call configure and close
        Hide
        Brock Noland added a comment -

        I think there are two issues above:

        1) Multiple input key value pairs
        2) Calling configure with an empty configuration object.

        I think the testing of multiple input key, value pairs should be handled in a fashion other than a static mapper. That is I would like to see MRUnit have the ability to process multiple input key value pairs in a single test with no statics or instance variable hacks. I created MRUNIT-64 to track this.

        It's possible to pass your own configuration object via

        {set,with}

        Configuration methods.

        Show
        Brock Noland added a comment - I think there are two issues above: 1) Multiple input key value pairs 2) Calling configure with an empty configuration object. I think the testing of multiple input key, value pairs should be handled in a fashion other than a static mapper. That is I would like to see MRUnit have the ability to process multiple input key value pairs in a single test with no statics or instance variable hacks. I created MRUNIT-64 to track this. It's possible to pass your own configuration object via {set,with} Configuration methods.

          People

          • Assignee:
            Brock Noland
            Reporter:
            E. Sammer
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development