Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1692

Remove TestStreamedMerge from the streaming tests

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: contrib/streaming
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Removed streaming testcase which tested non-existent functionality in Streaming.

      Description

      Currently the TestStreamedMerge is never run as a part of the streaming test suite, the code paths which were exercised by the test was removed in HADOOP-1315, so it is better to remove the testcase from the code base.

      1. ASF.LICENSE.NOT.GRANTED--MAPREDUCE-1692-1.patch
        10 kB
        Sreekanth Ramakrishnan
      2. ASF.LICENSE.NOT.GRANTED--MAPREDUCE-1692-1.patch
        11 kB
        Sreekanth Ramakrishnan
      3. ASF.LICENSE.NOT.GRANTED--patch-1692.txt
        12 kB
        Amareshwari Sriramadasu
      4. patch-1692-ydist.txt
        12 kB
        Vinod Kumar Vavilapalli

        Activity

        Hide
        Sreekanth Ramakrishnan added a comment -

        Attaching patch removing the test case.

        Show
        Sreekanth Ramakrishnan added a comment - Attaching patch removing the test case.
        Hide
        Sreekanth Ramakrishnan added a comment -

        Attaching patch correcting the build.xml in streaming.

        Show
        Sreekanth Ramakrishnan added a comment - Attaching patch correcting the build.xml in streaming.
        Hide
        Amareshwari Sriramadasu added a comment -

        +1 for removing the testcase.

        Show
        Amareshwari Sriramadasu added a comment - +1 for removing the testcase.
        Hide
        Hemanth Yamijala added a comment -

        The testcase (introduced in HADOOP-542) is quite obscure because there is no documentation about what it is intending to test. But trying to decipher:

        There are two options used in this testcase - mapsideoutput and inputtagged.

        Regarding mapsideoutput, what I could guess was it was using a streaming option that caused map outputs to be written to a side effect file and verified that the output from these files was as expected. In the test case, the file was set to be a socket or a local file whose URL was passed to the 'mapsideoutput' option in Streaming.

        As Sreekanth pointed out, this option is no longer supported, and hence the test code exercising this option is no longer needed.

        inputtagged is still getting added to streaming options in StreamJob. But it doesn't seem to be used anywhere. Can this be confirmed once again, please ? If confirmed to be unused, I am +1 for removing the test case. We must probably remove this option from being used in StreamJob as well, just so it doesn't cause confusion.

        If we stick to course to remove the test case, I suggest we remove the target test-unix in Streaming's build.xml. Because with the patch, it is now identical to the test target.

        Show
        Hemanth Yamijala added a comment - The testcase (introduced in HADOOP-542 ) is quite obscure because there is no documentation about what it is intending to test. But trying to decipher: There are two options used in this testcase - mapsideoutput and inputtagged. Regarding mapsideoutput, what I could guess was it was using a streaming option that caused map outputs to be written to a side effect file and verified that the output from these files was as expected. In the test case, the file was set to be a socket or a local file whose URL was passed to the 'mapsideoutput' option in Streaming. As Sreekanth pointed out, this option is no longer supported, and hence the test code exercising this option is no longer needed. inputtagged is still getting added to streaming options in StreamJob. But it doesn't seem to be used anywhere. Can this be confirmed once again, please ? If confirmed to be unused, I am +1 for removing the test case. We must probably remove this option from being used in StreamJob as well, just so it doesn't cause confusion. If we stick to course to remove the test case, I suggest we remove the target test-unix in Streaming's build.xml. Because with the patch, it is now identical to the test target.
        Hide
        Amareshwari Sriramadasu added a comment -

        inputtagged is still getting added to streaming options in StreamJob. But it doesn't seem to be used anywhere. Can this be confirmed once again, please ?

        I verified that inputtagged option is not used anywhere, not documented. It is just added to the options.

        Show
        Amareshwari Sriramadasu added a comment - inputtagged is still getting added to streaming options in StreamJob. But it doesn't seem to be used anywhere. Can this be confirmed once again, please ? I verified that inputtagged option is not used anywhere, not documented. It is just added to the options.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch incorporates Hemanth's suggestions.

        Show
        Amareshwari Sriramadasu added a comment - Patch incorporates Hemanth's suggestions.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12441925/patch-1692.txt
        against trunk revision 933441.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/115/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/115/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/115/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/115/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12441925/patch-1692.txt against trunk revision 933441. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/115/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/115/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/115/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/115/console This message is automatically generated.
        Hide
        Hemanth Yamijala added a comment -

        +1 for Amareshwari's last patch.

        Show
        Hemanth Yamijala added a comment - +1 for Amareshwari's last patch.
        Hide
        Hemanth Yamijala added a comment -

        I just committed this. Thanks, Sreekanth and Amareshwari !

        Show
        Hemanth Yamijala added a comment - I just committed this. Thanks, Sreekanth and Amareshwari !
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #289 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/289/)
        MAPREDUCE-1692. Removed unused testcase TestStreamedMerge. Contributed by Sreekanth Ramakrishnan and Amareshwari Sriramadasu.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #289 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/289/ ) MAPREDUCE-1692 . Removed unused testcase TestStreamedMerge. Contributed by Sreekanth Ramakrishnan and Amareshwari Sriramadasu.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Attaching patch for yahoo! dist on behalf of Amareshwari/Sreekanth. Not for commit here.

        Show
        Vinod Kumar Vavilapalli added a comment - Attaching patch for yahoo! dist on behalf of Amareshwari/Sreekanth. Not for commit here.

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Sreekanth Ramakrishnan
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development