Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1809

Ant build changes for Streaming system tests in contrib projects.

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.1
    • Component/s: build
    • Labels:
      None

      Description

      Implementing new target( test-system) in build-contrib.xml file for executing the system test that are in contrib projects. Also adding 'subant' target in aop.xml that calls the build-contrib.xml file for system tests.

      1. MAPREDUCE-1809.patch
        10 kB
        Vinay Kumar Thota
      2. MAPREDUCE-1809.patch
        11 kB
        Vinay Kumar Thota
      3. MAPREDUCE-1809.patch
        10 kB
        Vinay Kumar Thota
      4. MAPREDUCE-1809.patch
        10 kB
        Vinay Kumar Thota
      5. MAPREDUCE-1809.patch
        11 kB
        Vinay Kumar Thota
      6. MAPREDUCE-1809.patch
        11 kB
        Vinay Kumar Thota
      7. MAPREDUCE-1809.patch
        11 kB
        Vinay Kumar Thota
      8. MAPREDUCE-1809.patch
        16 kB
        Vinay Kumar Thota
      9. MAPREDUCE-1809.patch
        15 kB
        Vinay Kumar Thota
      10. 1809-ydist-security.patch
        10 kB
        Vinay Kumar Thota
      11. 1809-ydist-security.patch
        10 kB
        Vinay Kumar Thota

        Issue Links

          Activity

          Hide
          Vinay Kumar Thota added a comment -

          Please review initial patch and let me know your comments.

          Show
          Vinay Kumar Thota added a comment - Please review initial patch and let me know your comments.
          Hide
          Balaji Rajagopalan added a comment -

          For running the system test in contrib we are adding number of properties, I have question regarding some properties and what they are used for.

          <property name="test.all.tests.file" value="$

          {test.src.dir}

          /all-tests" />
          I think the above property is for the unit test.

          <property name="test.tools.input.dir" value="$

          {basedir}/src/test/tools/data"/>
          + <property name="c++.src" value="${basedir}

          /src/c++"/>
          + <property name="test.include" value="Test*"/>
          + <property name="c++.libhdfs.src" value="$

          {c++.src}

          /libhdfs"/>
          + <property name="test.build.data" value="$

          {build.test.system}/data"/>
          + <property name="test.cache.data" value="${build.test.system}

          /cache"/>
          + <property name="test.debug.data" value="$

          {build.test.system}

          /debug"/>

          Why do we need the above properties ?

          + <!--<target name="test-system" depends="ivy-retrieve-common"
          + description="Run system tests">-->

          The commented out code can be removed.

          Show
          Balaji Rajagopalan added a comment - For running the system test in contrib we are adding number of properties, I have question regarding some properties and what they are used for. <property name="test.all.tests.file" value="$ {test.src.dir} /all-tests" /> I think the above property is for the unit test. <property name="test.tools.input.dir" value="$ {basedir}/src/test/tools/data"/> + <property name="c++.src" value="${basedir} /src/c++"/> + <property name="test.include" value="Test*"/> + <property name="c++.libhdfs.src" value="$ {c++.src} /libhdfs"/> + <property name="test.build.data" value="$ {build.test.system}/data"/> + <property name="test.cache.data" value="${build.test.system} /cache"/> + <property name="test.debug.data" value="$ {build.test.system} /debug"/> Why do we need the above properties ? + <!--<target name="test-system" depends="ivy-retrieve-common" + description="Run system tests">--> The commented out code can be removed.
          Hide
          Vinay Kumar Thota added a comment -

          There is a common macro definition for executing the tests irrespective of unit or system tests.And this definition requires the all these properties to set the environment for executing the any kind of tests. So, I have used the same definition for executing the system tests that are under contrib projects.

          Show
          Vinay Kumar Thota added a comment - There is a common macro definition for executing the tests irrespective of unit or system tests.And this definition requires the all these properties to set the environment for executing the any kind of tests. So, I have used the same definition for executing the system tests that are under contrib projects.
          Hide
          Balaji Rajagopalan added a comment -

          I still think it is better to have seperate macros for unit and system test, instead of clubing everything together.

          Show
          Balaji Rajagopalan added a comment - I still think it is better to have seperate macros for unit and system test, instead of clubing everything together.
          Hide
          Vinay Kumar Thota added a comment -

          Removed the commented line as per your comment and attached the latest patch for yahoo distribution security branch.

          Show
          Vinay Kumar Thota added a comment - Removed the commented line as per your comment and attached the latest patch for yahoo distribution security branch.
          Hide
          Giridharan Kesavan added a comment -

          I coudnt apply the patch on 0.21 branch. Could you please recreate the patch ?

          Show
          Giridharan Kesavan added a comment - I coudnt apply the patch on 0.21 branch. Could you please recreate the patch ?
          Hide
          Vinay Kumar Thota added a comment -

          Patch for trunk.

          Show
          Vinay Kumar Thota added a comment - Patch for trunk.
          Hide
          Vinay Kumar Thota added a comment -

          Patch for yahoo dist security branch.

          Show
          Vinay Kumar Thota added a comment - Patch for yahoo dist security branch.
          Hide
          Vinay Kumar Thota added a comment -

          Re-created the patch from the source location.

          Show
          Vinay Kumar Thota added a comment - Re-created the patch from the source location.
          Hide
          Balaji Rajagopalan added a comment -

          +1

          Show
          Balaji Rajagopalan added a comment - +1
          Hide
          Konstantin Boudnik added a comment -

          looks Ok, please verify as usual

          Show
          Konstantin Boudnik added a comment - looks Ok, please verify as usual
          Hide
          Vinay Kumar Thota added a comment -

          added some attributes for excluding the system tests from the compilation of contribs.

          Show
          Vinay Kumar Thota added a comment - added some attributes for excluding the system tests from the compilation of contribs.
          Hide
          Vinay Kumar Thota added a comment -

          Updated the patch with latest changes.

          Show
          Vinay Kumar Thota added a comment - Updated the patch with latest changes.
          Hide
          Konstantin Boudnik added a comment -

          I'd suggest not to use build-fi directory anymore. There's that development MAPREDUCE-1897 is happening where all system-test related is getting compiled to build-fi/system in order to fix some issues with Herriot build

          Show
          Konstantin Boudnik added a comment - I'd suggest not to use build-fi directory anymore. There's that development MAPREDUCE-1897 is happening where all system-test related is getting compiled to build-fi/system in order to fix some issues with Herriot build
          Hide
          Vinay Kumar Thota added a comment -

          addressed the last comment.

          Show
          Vinay Kumar Thota added a comment - addressed the last comment.
          Hide
          Vinay Kumar Thota added a comment -

          Patch is generated with latest trunk.

          Show
          Vinay Kumar Thota added a comment - Patch is generated with latest trunk.
          Hide
          Vinay Kumar Thota added a comment -

          I ran the test-patch manually for make sure that patch is ok.

           
           +1 overall.
          
               +1 @author.  The patch does not contain any @author tags.
          
               +1 tests included.  The patch appears to include 19 new or modified tests.
          
               +1 javadoc.  The javadoc tool did not generate any warning messages.
          
               +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
          
               +1 findbugs.  The patch does not introduce any new Findbugs warnings.
          
               +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          
               +1 system tests framework.  The patch passed system tests framework compile.
          
          Show
          Vinay Kumar Thota added a comment - I ran the test-patch manually for make sure that patch is ok. +1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 19 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 system tests framework. The patch passed system tests framework compile.
          Hide
          Iyappan Srinivasan added a comment -

          +1

          Show
          Iyappan Srinivasan added a comment - +1
          Hide
          Amareshwari Sriramadasu added a comment -

          Can you run ant test-contrib with the patch? Also run test-patch and ant test-contrib in branch 0.21 also.

          Show
          Amareshwari Sriramadasu added a comment - Can you run ant test-contrib with the patch? Also run test-patch and ant test-contrib in branch 0.21 also.
          Hide
          Vinay Kumar Thota added a comment -

          I ran the test-patch manually against branch 0.21 for make sure that patch is ok.

           
          +1 overall.
          
               +1 @author.  The patch does not contain any @author tags.
          
               +1 tests included.  The patch appears to include 19 new or modified tests.
          
               +1 javadoc.  The javadoc tool did not generate any warning messages.
          
               +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
          
               +1 findbugs.  The patch does not introduce any new Findbugs warnings.
          
               +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          
               +1 system tests framework.  The patch passed system tests framework compile.
          

          Ran ant test-contrib with the patch and all the contrib tests executed successfully without any issues.

          [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 23.48 sec
          [junit] Running org.apache.hadoop.raid.TestReedSolomonDecoder
          [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 12.594 sec
          [junit] Running org.apache.hadoop.raid.TestReedSolomonEncoder
          [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.397 sec

          test:

          BUILD SUCCESSFUL
          Total time: 61 minutes 49 seconds

          Show
          Vinay Kumar Thota added a comment - I ran the test-patch manually against branch 0.21 for make sure that patch is ok. +1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 19 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 system tests framework. The patch passed system tests framework compile. Ran ant test-contrib with the patch and all the contrib tests executed successfully without any issues. [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 23.48 sec [junit] Running org.apache.hadoop.raid.TestReedSolomonDecoder [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 12.594 sec [junit] Running org.apache.hadoop.raid.TestReedSolomonEncoder [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.397 sec test: BUILD SUCCESSFUL Total time: 61 minutes 49 seconds
          Hide
          Amareshwari Sriramadasu added a comment -

          I just committed this. Thanks Vinay !

          Show
          Amareshwari Sriramadasu added a comment - I just committed this. Thanks Vinay !
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #556 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/556/)
          MAPREDUCE-1809. Ant build changes for Streaming system tests in contrib projects. Contributed by Vinay Kumar Thota

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #556 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/556/ ) MAPREDUCE-1809 . Ant build changes for Streaming system tests in contrib projects. Contributed by Vinay Kumar Thota
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )

            People

            • Assignee:
              Vinay Kumar Thota
              Reporter:
              Vinay Kumar Thota
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development