Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1084

Implementing aspects development and fault injeciton framework for MapReduce

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: build, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Similar to HDFS-435 and HADOOP-6204 this JIRA will track the introduction of injection framework for MapReduce.
      After HADOOP-6204 is in place this particular modification should be very trivial and would take importing (via svn:external) of src/test/build and some tweaking of the build.xml file

      1. mapreduce-1084-6-withRandom.patch
        20 kB
        Konstantin Boudnik
      2. mapreduce-1084-6.patch
        12 kB
        Sreekanth Ramakrishnan
      3. mapreduce-1084-5.patch
        12 kB
        Sreekanth Ramakrishnan
      4. mapreduce-1084-final.patch
        11 kB
        Sreekanth Ramakrishnan
      5. mapreduce-1084-3.patch
        11 kB
        Sreekanth Ramakrishnan
      6. mapreduce-1084-2.patch
        11 kB
        Sreekanth Ramakrishnan
      7. mapreduce-1084-1-withoutsvnexternals.patch
        6 kB
        Sreekanth Ramakrishnan
      8. mapreduce-1084-1.patch
        6 kB
        Sreekanth Ramakrishnan

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching the patch implementing the fault injection in mapreduce project.

          There are two patches with svn external and without svn external. Svn external patch when applied over workspace does not create the appropriate folder structure with links even tho' the property and folder is added into version control.

          Show
          Sreekanth Ramakrishnan added a comment - Attaching the patch implementing the fault injection in mapreduce project. There are two patches with svn external and without svn external. Svn external patch when applied over workspace does not create the appropriate folder structure with links even tho' the property and folder is added into version control.
          Hide
          Konstantin Boudnik added a comment -

          svn external patch is better to be abandoned because vanilla git doesn't understand svn:externals. So, lets consider the patch without externals as the only alternative at the moment.

          Few comments:

          • it seems like patch is missing src/test/aop/build/aop.xml file
          • target name jar-mapred-test-fault-inject needs to be changed to jar-mapred-fault-inject to be compliant with Common and HDFS names
          Show
          Konstantin Boudnik added a comment - svn external patch is better to be abandoned because vanilla git doesn't understand svn:externals. So, lets consider the patch without externals as the only alternative at the moment. Few comments: it seems like patch is missing src/test/aop/build/aop.xml file target name jar-mapred-test-fault-inject needs to be changed to jar-mapred-fault-inject to be compliant with Common and HDFS names
          Hide
          Sreekanth Ramakrishnan added a comment -

          Added the target jar-test-fault-inject which points to jar-mapred-test-fault-inject same way as it is implemented in HDFS-703.

          Including the src/test/aop/build/aop.xml from commons-trunk.

          Show
          Sreekanth Ramakrishnan added a comment - Added the target jar-test-fault-inject which points to jar-mapred-test-fault-inject same way as it is implemented in HDFS-703 . Including the src/test/aop/build/aop.xml from commons-trunk.
          Hide
          Konstantin Boudnik added a comment -

          Why jar-mapred-test-fault-inject is needed at all?

          Show
          Konstantin Boudnik added a comment - Why jar-mapred-test-fault-inject is needed at all?
          Hide
          Sreekanth Ramakrishnan added a comment -

          Renaming jar-mapred-test-fault-inject to jar-test-fault-inject.

          The reason why jar-mapred-test-fault-inject was implemented was to follow the same standard as in HDFS-703

          Show
          Sreekanth Ramakrishnan added a comment - Renaming jar-mapred-test-fault-inject to jar-test-fault-inject . The reason why jar-mapred-test-fault-inject was implemented was to follow the same standard as in HDFS-703
          Hide
          Sreekanth Ramakrishnan added a comment -

          Running thro' HUDSON.

          Show
          Sreekanth Ramakrishnan added a comment - Running thro' HUDSON.
          Hide
          Konstantin Boudnik added a comment -

          Renaming jar-mapred-test-fault-inject to jar-test-fault-inject.

          Sorry for the confusion, Sreekanth - you're right. I was pretty sure that HDFS has 'jar-test-fault-inject'. I guess I'm withdrawing my comment about this.

          Show
          Konstantin Boudnik added a comment - Renaming jar-mapred-test-fault-inject to jar-test-fault-inject. Sorry for the confusion, Sreekanth - you're right. I was pretty sure that HDFS has 'jar-test-fault-inject'. I guess I'm withdrawing my comment about this.
          Hide
          Konstantin Boudnik added a comment -

          +1 on mapreduce-1084-2.patch - looks good.

          Show
          Konstantin Boudnik added a comment - +1 on mapreduce-1084-2.patch - looks good.
          Hide
          Sreekanth Ramakrishnan added a comment -

          I have uploaded both patches, the patch number 3 has the two targets:

          jar-test-fault-inject
          -> jar-mapred-test-fault-inject

          Patch 4 has just jar-test-fault-inject.

          Am I required to upload a new patch for committing which would be same as patch number 3?

          Show
          Sreekanth Ramakrishnan added a comment - I have uploaded both patches, the patch number 3 has the two targets: jar-test-fault-inject -> jar-mapred-test-fault-inject Patch 4 has just jar-test-fault-inject . Am I required to upload a new patch for committing which would be same as patch number 3?
          Hide
          Sreekanth Ramakrishnan added a comment -

          Sorry for confusion in above comment it is patch 2 and patch 3 instead of 3 and 4. Sorry for the confusion.

          Show
          Sreekanth Ramakrishnan added a comment - Sorry for confusion in above comment it is patch 2 and patch 3 instead of 3 and 4. Sorry for the confusion.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427286/mapreduce-1084-3.patch
          against trunk revision 888144.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/171/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/171/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/171/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/171/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427286/mapreduce-1084-3.patch against trunk revision 888144. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/171/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/171/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/171/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/171/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          To clear off the confusions: I vote +1 for patch-2 without 'jar-test-fault-inject' which I've requested by mistake.

          Show
          Konstantin Boudnik added a comment - To clear off the confusions: I vote +1 for patch-2 without 'jar-test-fault-inject' which I've requested by mistake.
          Hide
          Sreekanth Ramakrishnan added a comment -

          Reattaching mapreduce-1084-2.patch as final patch.

          Show
          Sreekanth Ramakrishnan added a comment - Reattaching mapreduce-1084-2.patch as final patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427432/mapreduce-1084-final.patch
          against trunk revision 888431.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/175/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/175/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/175/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/175/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427432/mapreduce-1084-final.patch against trunk revision 888431. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/175/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/175/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/175/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/175/console This message is automatically generated.
          Hide
          Sreekanth Ramakrishnan added a comment -

          Cancelling patch, due to hudson issue.

          Show
          Sreekanth Ramakrishnan added a comment - Cancelling patch, due to hudson issue.
          Hide
          Sreekanth Ramakrishnan added a comment -

          Found an issue with contrib project vaidya, which accidentally re includes the top level build.xml, this was causing the compilation of the contrib project itself breaking. I have removed the change which was put in by MAPREDUCE-676

          The core test failure is not related to the patch as the patch does not change any java code nor modify the existing classpaths which junit runner uses.

          Show
          Sreekanth Ramakrishnan added a comment - Found an issue with contrib project vaidya, which accidentally re includes the top level build.xml, this was causing the compilation of the contrib project itself breaking. I have removed the change which was put in by MAPREDUCE-676 The core test failure is not related to the patch as the patch does not change any java code nor modify the existing classpaths which junit runner uses.
          Hide
          Konstantin Boudnik added a comment -

          Technically it isn't a good idea to mix together two different fixes in one patch. However, in this case I suppose it is Ok because otherwise the build will be broken.

          +1 patch looks good.

          Show
          Konstantin Boudnik added a comment - Technically it isn't a good idea to mix together two different fixes in one patch. However, in this case I suppose it is Ok because otherwise the build will be broken. +1 patch looks good.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427461/mapreduce-1084-5.patch
          against trunk revision 889085.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/182/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427461/mapreduce-1084-5.patch against trunk revision 889085. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/182/console This message is automatically generated.
          Hide
          Sreekanth Ramakrishnan added a comment -

          Patch merged with latest trunk.

          Show
          Sreekanth Ramakrishnan added a comment - Patch merged with latest trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427580/mapreduce-1084-6.patch
          against trunk revision 889085.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/183/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/183/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/183/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/183/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427580/mapreduce-1084-6.patch against trunk revision 889085. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/183/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/183/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/183/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/183/console This message is automatically generated.
          Hide
          Sreekanth Ramakrishnan added a comment -

          The test case failures for this test seems to be caused by MAPREDUCE-1275

          Locally there were no test case failures of any sort.

          Show
          Sreekanth Ramakrishnan added a comment - The test case failures for this test seems to be caused by MAPREDUCE-1275 Locally there were no test case failures of any sort.
          Hide
          Konstantin Boudnik added a comment -

          Great. Looks like it is good to go. I'll commit it tomorrow morning if there no one else around to do this earlier

          Show
          Konstantin Boudnik added a comment - Great. Looks like it is good to go. I'll commit it tomorrow morning if there no one else around to do this earlier
          Hide
          Konstantin Boudnik added a comment -

          While going through the last round of the patch's testing I've noticed that it misses all files related to random fault injection, i.e. ProbabilityModel, etc. In order to speed up the patch delivery I'm going to submit the new one including these files and run it through test-patch process one more time before committing.

          Show
          Konstantin Boudnik added a comment - While going through the last round of the patch's testing I've noticed that it misses all files related to random fault injection, i.e. ProbabilityModel, etc. In order to speed up the patch delivery I'm going to submit the new one including these files and run it through test-patch process one more time before committing.
          Hide
          Konstantin Boudnik added a comment -

          Adding probability model and config management. Also, adding missed modification to build.xml to include default fi-site.xml config file to instrumented jar file.

          Show
          Konstantin Boudnik added a comment - Adding probability model and config management. Also, adding missed modification to build.xml to include default fi-site.xml config file to instrumented jar file.
          Hide
          Konstantin Boudnik added a comment -

          Need to run new patch through test-patch process again.

          Show
          Konstantin Boudnik added a comment - Need to run new patch through test-patch process again.
          Hide
          Konstantin Boudnik added a comment -

          resubmitting

          Show
          Konstantin Boudnik added a comment - resubmitting
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427800/mapreduce-1084-6-withRandom.patch
          against trunk revision 889786.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          -1 release audit. The applied patch generated 162 release audit warnings (more than the trunk's current 161 warnings).

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/322/testReport/
          Release audit warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/322/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/322/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/322/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/322/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427800/mapreduce-1084-6-withRandom.patch against trunk revision 889786. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 162 release audit warnings (more than the trunk's current 161 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/322/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/322/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/322/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/322/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/322/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          The audit warning is cause by the absence of Apache's license boiler plate in fi-site.xml file which isn't needed.
          Test failures are again caused by MAPREDUCE-1275.

          Show
          Konstantin Boudnik added a comment - The audit warning is cause by the absence of Apache's license boiler plate in fi-site.xml file which isn't needed. Test failures are again caused by MAPREDUCE-1275 .
          Hide
          Konstantin Boudnik added a comment -

          I've committed the latest patch with my fixes. Thanks Sreekanth!

          Show
          Konstantin Boudnik added a comment - I've committed the latest patch with my fixes. Thanks Sreekanth!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/ )

            People

            • Assignee:
              Sreekanth Ramakrishnan
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development