MRUnit
  1. MRUnit
  2. MRUNIT-129

Key object re-use in Reducer is inconsistent with MapReduce behaviour (new API)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 1.1.0
    • Labels:
      None

      Description

      Follow on issue from MRUNIT-127, also relating to grouping comparators.

      In the new MapReduce API Reducer's reduce method, you are passed a reference to a key and an Iterable for the values. What's not particularly clear is that as you iterate through the values, the value of the key is also changing silently. This is not usually noticeable, since the keys that were grouped often have the same value anyway, but with a custom grouping comparator this isn't necessarily the case.

      This is different to the old API, where you would get the first key in the group and this wouldn't change as you iterate over the values.

      MRUnit uses the "old API" style behaviour for both old and new APIs, so some unit tests may not give a consistent result to an actual mapreduce job.

      1. JIRA_129.patch
        55 kB
        Paulin Sanselme
      2. MRUNIT-129.unittest.patch
        4 kB
        Dave Beech

        Activity

        Dave Beech created issue -
        Dave Beech made changes -
        Field Original Value New Value
        Attachment MRUNIT-129.unittest.patch [ 12539163 ]
        Paulin Sanselme made changes -
        Attachment JIRA_129.patch [ 12576602 ]
        Paulin Sanselme made changes -
        Attachment JIRA_129.patch [ 12576602 ]
        Paulin Sanselme made changes -
        Attachment JIRA_129.patch [ 12576805 ]
        Dave Beech made changes -
        Fix Version/s 1.1.0 [ 12324262 ]
        Brock Noland made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Assignee Paulin Sanselme [ paulin sanselme ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Paulin Sanselme
            Reporter:
            Dave Beech
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development