MRUnit
  1. MRUnit
  2. MRUNIT-48

Test and Deprecate Driver.{*OutputFromString,*InputFromString} Methods

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Labels:
      None

      Description

      We should test a driver that only deals with, say LongWritable, to ensure the runTest() method fails when we use the Driver.{*OutputFromString,*InputFromString} Methods.

      Additionally, I think we should deprecate these methods since they don't play well with our generic interfaces.

      1. MRUNIT-48-1.patch
        18 kB
        Jim Donofrio

        Activity

        Hide
        Jim Donofrio added a comment -

        I am not sure what you mean by this, do you mean only pass in a key with no tab delimited value so that the input or output method doesnt add a input or output?

        Show
        Jim Donofrio added a comment - I am not sure what you mean by this, do you mean only pass in a key with no tab delimited value so that the input or output method doesnt add a input or output?
        Hide
        Brock Noland added a comment -

        If a MRUnit test is using say LongWritable as the key, it's still possible to use the setOutputFromString methods. The setOutputFromString methods have no way of knowing the key should be a LongWritable. As such, I think we should 1) deprecate these methods 2) write a unit test to confirm that if the key is a non-Text type, MRUnit actually fails somewhere with a good error message.

        Show
        Brock Noland added a comment - If a MRUnit test is using say LongWritable as the key, it's still possible to use the setOutputFromString methods. The setOutputFromString methods have no way of knowing the key should be a LongWritable. As such, I think we should 1) deprecate these methods 2) write a unit test to confirm that if the key is a non-Text type, MRUnit actually fails somewhere with a good error message.
        Hide
        Jim Donofrio added a comment -

        deprecated all add/with Output/Input FromString methods
        added test to confirm that using InputFromString with a mapper or reducer that doesnt take a Text will throw a ClassCastException which is probably good enough
        However, OutputFromString is confusing because the log output will print the exact same output which isnt equal because the classes dont match. Added to additional else if statements in TestDriver to throw/log a more meaningful error such as "Missing expected output (a, 1): Mismatch in value class: expected: class org.apache.hadoop.io.Text actual: class org.apache.hadoop.io.LongWritable" instead of seeing:
        Received unexpected output (a, 1)
        MIssing expected output (a, 1)

        Only test these cases in mrunit.TestMapDriver because all validation is done through the common TestDriver class

        After a release are we going to delete these methods, they seem to add more confusion than they are worth?

        Show
        Jim Donofrio added a comment - deprecated all add/with Output/Input FromString methods added test to confirm that using InputFromString with a mapper or reducer that doesnt take a Text will throw a ClassCastException which is probably good enough However, OutputFromString is confusing because the log output will print the exact same output which isnt equal because the classes dont match. Added to additional else if statements in TestDriver to throw/log a more meaningful error such as "Missing expected output (a, 1): Mismatch in value class: expected: class org.apache.hadoop.io.Text actual: class org.apache.hadoop.io.LongWritable" instead of seeing: Received unexpected output (a, 1) MIssing expected output (a, 1) Only test these cases in mrunit.TestMapDriver because all validation is done through the common TestDriver class After a release are we going to delete these methods, they seem to add more confusion than they are worth?
        Hide
        Jim Donofrio added a comment -

        Committed in 1297378

        Show
        Jim Donofrio added a comment - Committed in 1297378
        Hide
        Brock Noland added a comment -

        Cool! Yes, I think we should remove then in the next major release after 1.0.

        Show
        Brock Noland added a comment - Cool! Yes, I think we should remove then in the next major release after 1.0.

          People

          • Assignee:
            Jim Donofrio
            Reporter:
            Brock Noland
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development