Issue Details (XML | Word | Printable)

Key: HDFS-475
Type: Sub-task Sub-task
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: Konstantin Boudnik
Reporter: Konstantin Boudnik
Votes: 0
Watchers: 3
Operations

If you were logged in you would be able to see more operations.
Hadoop HDFS
HDFS-435

Create a separate targets for fault injection related test and jar files creation files

Created: 08/Jul/09 06:29 PM   Updated: 17/Aug/09 10:35 PM
Return to search
Component/s: build
Affects Version/s: 0.21.0
Fix Version/s: 0.21.0

Time Tracking:
Not Specified

File Attachments:
  Size
Text File Licensed for inclusion in ASF works h475_20090716ignore.patch 2009-07-17 12:47 AM Tsz Wo (Nicholas), SZE 0.5 kB
Text File Licensed for inclusion in ASF works HDFS-475.patch 2009-07-16 06:27 PM Konstantin Boudnik 8 kB
Text File Licensed for inclusion in ASF works HDFS-475.patch 2009-07-16 06:00 PM Konstantin Boudnik 8 kB
Text File Licensed for inclusion in ASF works HDFS-475.patch 2009-07-14 12:16 AM Konstantin Boudnik 8 kB
Text File Licensed for inclusion in ASF works HDFS-475.patch 2009-07-13 06:53 PM Konstantin Boudnik 8 kB
Text File Licensed for inclusion in ASF works HDFS-475.patch 2009-07-08 08:44 PM Konstantin Boudnik 3 kB
Issue Links:
Reference

Hadoop Flags: Reviewed
Resolution Date: 17/Jul/09 12:48 AM


 Description  « Hide
Current implementation of the FI framework allows to mix faults into production classes, e.g. into build/ folder.
Although the default probability level is set to zero it doesn't look clean and might potentially over complicate the build and release process.

FI related targets are better be logically and physically separated, e.g. to put instrumented artifacts into a separate folder, say, build-fi/



 All   Comments   Work Log   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Konstantin Boudnik added a comment - 08/Jul/09 08:50 PM
This patch contains the following modifications of build.xml file:
  • new tar 'jar-fi' is added to create devepment jar file with the classes instrumented by injected faults. The name of this new tar file differs from normal dev.jar and looks like hadoop-hdfs-0.21.0-dev-fi.jar
  • target 'jar' has been modified to depend on 'clean' in order to guarantee that only non fault-injected classes are built and included into normal development jar file. Performance implications of additional cleaning seem to be very low if any.

Target 'tar' hasn't been altered for I don't see why FI'ed stuff has to be included into the release tarball.


Konstantin Boudnik added a comment - 08/Jul/09 09:12 PM
The first bullet point has to be read as:
  • new target 'jar-fi' is added to create devepment jar file with the classes
    instrumented by injected faults. The name of this new tar file differs from
    normal dev.jar and looks like hadoop-hdfs-0.21.0-dev-fi.jar


With best regards,
Konstantin Boudnik (aka Cos)

Yahoo! Grid Computing
+1 (408) 349-4049

2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622
Attention! Streams of consciousness are disallowed


Hadoop QA added a comment - 08/Jul/09 09:35 PM
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12412910/HDFS-475.patch
against trunk revision 792310.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/9/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/9/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/9/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/9/console

This message is automatically generated.


Konstantin Boudnik added a comment - 08/Jul/09 09:43 PM
The test failure is unrelated to the patch for it has been broken from the very first build of HDFS
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/0/

Tsz Wo (Nicholas), SZE added a comment - 08/Jul/09 11:22 PM
+1 patch looks good. I will wait for one, two days before committing this.

Konstantin Boudnik added a comment - 13/Jul/09 06:53 PM
The patch adds the following high level targets:
  • jar-fi Make hadoop-fi.jar
  • jar-hdfs-test-fi Make hadoop-test-fi.jar
  • jar-hdfswithmr-test-fi Make hadoop-test-fi.jar
  • jar-test-fi Make hadoop-test.jar
  • run-test-hdfs-fi Run FI related hdfs tests
  • run-test-hdfs-with-mr-fi Run hdfs FI related unit tests that require mapred

Tsz Wo (Nicholas), SZE added a comment - 13/Jul/09 11:35 PM
In HDFS-483, I implemented some tests which use fi. I have to change the test target as following in order to run the tests. Do you want to add them to your patch?
@@ -318,7 +318,7 @@
   <!-- Weaving aspects in place 
   	Later on one can run 'ant jar' to create Hadoop jar file with instrumented classes
   -->
-  <target name="injectfaults" depends="compile" description="Weaves aspects into precomplied HDFS classes">
+  <target name="injectfaults" depends="compile, compile-hdfs-test" description="Weaves aspects into precomplied HDFS classes">
     <!-- AspectJ task definition -->
     <taskdef resource="org/aspectj/tools/ant/taskdefs/aspectjTaskdefs.properties">
       <classpath>
@@ -335,7 +335,7 @@
       target="${javac.version}"
       source="${javac.version}"
       deprecation="${javac.deprecation}">
-        <classpath refid="classpath" />
+        <classpath refid="test.classpath" />
     </iajc>
     <echo message="Weaving of aspects is finished"/>
   </target>
@@ -500,10 +500,14 @@
       <batchtest todir="${test.build.dir}" unless="testcase">
         <fileset dir="${test.src.dir}/hdfs"
            includes="**/${test.include}.java"
-     excludes="**/${test.exclude}.java" />
+           excludes="**/${test.exclude}.java" />
+        <fileset dir="${test.src.dir}/aop"
+           includes="**/${test.include}.java"
+           excludes="**/${test.exclude}.java" />
       </batchtest>
       <batchtest todir="${test.build.dir}" if="testcase">
         <fileset dir="${test.src.dir}/hdfs" includes="**/${testcase}.java"/>
+        <fileset dir="${test.src.dir}/aop" includes="**/${testcase}.java"/>
       </batchtest>
     </junit>
     <antcall target="checkfailure"/>

Tsz Wo (Nicholas), SZE added a comment - 13/Jul/09 11:37 PM
Looks like that your patch also fixes HDFS-476. Is it the case?

Konstantin Boudnik added a comment - 14/Jul/09 12:16 AM
The latest patch version includes Nicholas' modifications to include src/test/aop as a source for additional tests (FI specific ones). The only modification I had to make in suggested addition is to replace the dependency of 'injectfaults' target from 'compile' to 'compile-core'.

It doesn't look to me that contrib and ant-tasks are related to the FI anyhow. Please correct me if I'm wrong.


Konstantin Boudnik added a comment - 14/Jul/09 12:22 AM
HDFS-476 will be fixed by this patch as well although in a slightly different manner. Now in order to run tests with injected faults one doesn't need to run 'injectfaults' target separately. It could be taken care off in a single command:
ant run-hdfs-test-fi -DTestFiXxx

I believe this is the essence of HDFS-476.

Also, this new version of the patch provides new targets to create dev. and test jar files with included FI instrumentation.

Please be advised, that Hudson's test-patch output can't be provided for this patch, because HDFS's Hudson doesn't run new targets right now.


Konstantin Boudnik added a comment - 16/Jul/09 06:00 PM
Removing unnecessary extra-indentation

Konstantin Boudnik added a comment - 16/Jul/09 06:27 PM
Including modifications of the target names as mentioned by HDFS-476

Konstantin Boudnik added a comment - 16/Jul/09 06:32 PM
Current list of fault injection targets looks like as follows:
jar-fault-inject                    Make hadoop-fi.jar
jar-hdfs-test-fault-inject          Make hadoop-test-fi.jar
jar-hdfswithmr-test-fault-inject    Make hadoop-hdfswithmr-test-fi.jar
jar-test-fault-inject               Make hadoop-test.jar files
run-test-hdfs-fault-inject          Run Fault Injection related hdfs tests
run-test-hdfs-with-mr-fault-inject  Run hdfs Fault Injection related unit tests that require mapred

Tsz Wo (Nicholas), SZE added a comment - 17/Jul/09 12:28 AM
+1 patch looks good.

Tried all the ant targets listed above. It works fine.


Tsz Wo (Nicholas), SZE added a comment - 17/Jul/09 12:37 AM
Also tried
ant run-test-hdfs-fault-inject -Dtestcase=TestFiDataTransferProtocol

with the patch posted in HDFS-483. It works.

Note that the patch changes the ant target name as shown below.

-  <target name="injectfaults" depends="compile" description="Weaves aspects into precomplied HDFS classes">
+  <target name="compile-fault-inject" depends="compile-core, compile-hdfs-test">

"injectfaults" was introduced by HDFS-436 which is committed to 0.21. I think this is not an incompatible change since 0.21 is not yet released.


Tsz Wo (Nicholas), SZE added a comment - 17/Jul/09 12:47 AM
h475_20090716ignore.patch: add build-fi to the ignore lists.

Tsz Wo (Nicholas), SZE added a comment - 17/Jul/09 12:48 AM
I have committed this. Thanks, Cos!

Hudson added a comment - 17/Jul/09 02:06 PM
Integrated in Hadoop-Hdfs-trunk #25 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/25/)
. Add new ant targets for fault injection jars and tests. Contributed by Konstantin Boudnik