Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1369

JUnit tests should never depend on anything in conf

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The recent change to mapred-queues.xml that causes many mapreduce tests to break unless you delete conf/mapred-queues.xml out of your build tree is bad. We need to make sure that nothing in conf is used in the unit tests. One potential solution is to copy the templates into build/test/conf and use that instead.

      1. MAPREDUCE-1369.patch
        1 kB
        Anatoli Fomenko

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/ )
          Hide
          Konstantin Boudnik added a comment -

          Sounds like a right idea to me.

          Show
          Konstantin Boudnik added a comment - Sounds like a right idea to me.
          Hide
          Amareshwari Sriramadasu added a comment -

          This is still a problem with contrib tests. Should we do the same thing in each contrib build.xml?
          Can we re-use this code?

          Show
          Amareshwari Sriramadasu added a comment - This is still a problem with contrib tests. Should we do the same thing in each contrib build.xml? Can we re-use this code?
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/221/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/221/ )
          Hide
          Konstantin Boudnik added a comment -

          I've just committed this. Thank you Anatoli!

          Show
          Konstantin Boudnik added a comment - I've just committed this. Thank you Anatoli!
          Hide
          Konstantin Boudnik added a comment -

          +1 patch looks good. It is essentially the same as HADOOP-6374 and should be applied to both trunk and 0.21

          Show
          Konstantin Boudnik added a comment - +1 patch looks good. It is essentially the same as HADOOP-6374 and should be applied to both trunk and 0.21
          Hide
          Anatoli Fomenko added a comment -

          Please review the submitted patch. Thanks.

          Show
          Anatoli Fomenko added a comment - Please review the submitted patch. Thanks.
          Hide
          Anatoli Fomenko added a comment -

          Please review the submitted patch. Thanks.

          Show
          Anatoli Fomenko added a comment - Please review the submitted patch. Thanks.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12429868/MAPREDUCE-1369.patch
          against trunk revision 897118.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/372/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/372/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/372/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/372/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429868/MAPREDUCE-1369.patch against trunk revision 897118. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/372/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/372/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/372/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/372/console This message is automatically generated.
          Hide
          Anatoli Fomenko added a comment -

          Please review the submitted patch.

          Some details of this patch:

          1. In Map/Reduce build.xml file test.classpath includes classpath which, in turn, includes

            <property name="conf.dir" value="${basedir}/conf"/>
          

          2. The template files are copied to conf.dir by the following code:

            <copy todir="${conf.dir}" verbose="true">
              <fileset dir="${conf.dir}" includes="**/*.template"/>
              <mapper type="glob" from="*.template" to="*"/>
            </copy>
          

          3. Per the initial fix suggestion, a new property is introduced:

            <property name="test.conf.dir" value="${build.dir}/test/conf"/>
          

          It replaces the conf.dir in the test.classpath:

            <path id="test.classpath">
              <pathelement location="${test.build.extraconf}"/>
              <pathelement location="${test.core.build.classes}" />
              <pathelement location="${test.src.dir}"/>
              <pathelement location="${build.dir}"/>
              <pathelement location="${build.examples}"/>
              <pathelement location="${build.tools}"/>
              <pathelement path="${clover.jar}"/>
            - <path refid="classpath"/>
            + <pathelement location="${build.classes}"/>
            + <pathelement location="${test.conf.dir}"/>
            + <path refid="ivy-common.classpath"/>
              <pathelement location="${test.mapred.build.classes}" />
              <path refid="ivy-test.classpath"/>
            </path>
          

          4. The following code is added to copy templates from conf.dit to test.conf.dir:

            <copy todir="${test.conf.dir}" verbose="true">
              <fileset dir="${conf.dir}" includes="**/*.template"/>
              <mapper type="glob" from="*.template" to="*"/>
            </copy>
          
          Show
          Anatoli Fomenko added a comment - Please review the submitted patch. Some details of this patch: 1. In Map/Reduce build.xml file test.classpath includes classpath which, in turn, includes <property name="conf.dir" value="${basedir}/conf"/> 2. The template files are copied to conf.dir by the following code: <copy todir="${conf.dir}" verbose="true"> <fileset dir="${conf.dir}" includes="**/*.template"/> <mapper type="glob" from="*.template" to="*"/> </copy> 3. Per the initial fix suggestion, a new property is introduced: <property name="test.conf.dir" value="${build.dir}/test/conf"/> It replaces the conf.dir in the test.classpath: <path id="test.classpath"> <pathelement location="${test.build.extraconf}"/> <pathelement location="${test.core.build.classes}" /> <pathelement location="${test.src.dir}"/> <pathelement location="${build.dir}"/> <pathelement location="${build.examples}"/> <pathelement location="${build.tools}"/> <pathelement path="${clover.jar}"/> - <path refid="classpath"/> + <pathelement location="${build.classes}"/> + <pathelement location="${test.conf.dir}"/> + <path refid="ivy-common.classpath"/> <pathelement location="${test.mapred.build.classes}" /> <path refid="ivy-test.classpath"/> </path> 4. The following code is added to copy templates from conf.dit to test.conf.dir: <copy todir="${test.conf.dir}" verbose="true"> <fileset dir="${conf.dir}" includes="**/*.template"/> <mapper type="glob" from="*.template" to="*"/> </copy>

            People

            • Assignee:
              Anatoli Fomenko
              Reporter:
              Anatoli Fomenko
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development