Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-703

Replace current fault injection implementation with one from Common

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      After HADOOP-6204 has been implemented HDFS doesn't need to have its separate implementation of fault injection framework. Instead it has to reuse one from Common.

      1. HDFS-703.patch
        9 kB
        Konstantin Boudnik
      2. HDFS-703.patch
        9 kB
        Konstantin Boudnik
      3. HDFS-703.patch
        9 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #72 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/72/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #72 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/72/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #136 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/136/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #136 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/136/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #100 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/100/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #100 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/100/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #98 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/98/)
          . Removing svn external and adding build/aop.xml file physically to fix incompatibility with git (cos)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #98 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/98/ ) . Removing svn external and adding build/aop.xml file physically to fix incompatibility with git (cos)
          Hide
          Konstantin Boudnik added a comment -

          Well, mine too Glad everything is back to normal again. Thanks all for the patience.

          As for code divergence: I really hope we'd be able to switch to Maven some day which will address some of these issues.

          Show
          Konstantin Boudnik added a comment - Well, mine too Glad everything is back to normal again. Thanks all for the patience. As for code divergence: I really hope we'd be able to switch to Maven some day which will address some of these issues.
          Hide
          Todd Lipcon added a comment -

          I agree the code duplication is annoying, but post-split I think it makes sense. If common makes a change, we expect them to test it in common but might not notice if the change breaks mapreduce and HDFS. Down the line the build processes might diverge more, in which case having separate copies is good.

          Thanks for checking in the file. My trunk builds again

          Show
          Todd Lipcon added a comment - I agree the code duplication is annoying, but post-split I think it makes sense. If common makes a change, we expect them to test it in common but might not notice if the change breaks mapreduce and HDFS. Down the line the build processes might diverge more, in which case having separate copies is good. Thanks for checking in the file. My trunk builds again
          Hide
          Konstantin Boudnik added a comment -

          Ok, I've removed the svn:externals from the directory in question and have checked-in the build/aop.xml file physically. It will fix the git problem.

          Show
          Konstantin Boudnik added a comment - Ok, I've removed the svn:externals from the directory in question and have checked-in the build/aop.xml file physically. It will fix the git problem.
          Hide
          Konstantin Boudnik added a comment -

          If svn:externals seems to be an inappropriate - I don't like them myself - I can simply checkin this file into all subprojects, which is ugly too but at least will allow to keep the injection stuff encapsulated.

          Show
          Konstantin Boudnik added a comment - If svn:externals seems to be an inappropriate - I don't like them myself - I can simply checkin this file into all subprojects, which is ugly too but at least will allow to keep the injection stuff encapsulated.
          Hide
          Konstantin Boudnik added a comment -

          Well, the point of having this file as external was exactly to avoid checking it in. Our build system has too many build code duplication all over the place. I didn't want to increase the amount of duplicated code by adding a relatively big chunk of exactly same xml into three different classes.

          We already have the same SVN-git synchronization problem with test-patch script. I see two ways of solving this problem:

          • have the svn:externals represented by symlinks in git repo (there are ready solutions for this)
          • checkin files represented by svn:externals into git repositories (might have some strange side effects, I think)
          Show
          Konstantin Boudnik added a comment - Well, the point of having this file as external was exactly to avoid checking it in. Our build system has too many build code duplication all over the place. I didn't want to increase the amount of duplicated code by adding a relatively big chunk of exactly same xml into three different classes. We already have the same SVN-git synchronization problem with test-patch script. I see two ways of solving this problem: have the svn:externals represented by symlinks in git repo (there are ready solutions for this) checkin files represented by svn:externals into git repositories (might have some strange side effects, I think)
          Hide
          Eli Collins added a comment -

          Hey Konstantin,

          Can we check that file in? I don't use svn and so can't build if I sync past this change.

          Thanks,
          Eli

          Show
          Eli Collins added a comment - Hey Konstantin, Can we check that file in? I don't use svn and so can't build if I sync past this change. Thanks, Eli
          Hide
          Todd Lipcon added a comment -

          This breaks the build for those of us using git-svn (or the git.apache.org mirror). This seems like a complete blocker - most of the hadoop developers I've spoken with use git-svn for their working trees.

          Would like to propose this be reverted, or the file duplicated into the local tree.

          Show
          Todd Lipcon added a comment - This breaks the build for those of us using git-svn (or the git.apache.org mirror). This seems like a complete blocker - most of the hadoop developers I've spoken with use git-svn for their working trees. Would like to propose this be reverted, or the file duplicated into the local tree.
          Hide
          Konstantin Boudnik added a comment -

          Here's what I've just did:

            % svn co https://svn.apache.org/repos/asf/hadoop/hdfs/trunk new-hdfs-workspace
          ...
          Fetching external item into '123/src/test/aop/build'
          A    new-hdfs-workspace/src/test/aop/build/aop.xml
          Checked out external at revision 832876.
          ...
          

          As you can see the referred aop.xml file is imported from Common project through svn:externals

            % cd new-hdfs-workspace
            % ls -l src/test/aop/build
          aop.xml
            % ant injectfaults
          ...
          compile-fault-inject:
               [echo] Start weaving aspects in place
               [echo] Weaving of aspects is finished
          
          BUILD SUCCESSFUL
          Total time: 1 minute 11 seconds
          

          So everything seems to be working properly. Hope it helps to resolve the issue.

          Show
          Konstantin Boudnik added a comment - Here's what I've just did: % svn co https://svn.apache.org/repos/asf/hadoop/hdfs/trunk new-hdfs-workspace ... Fetching external item into '123/src/test/aop/build' A new-hdfs-workspace/src/test/aop/build/aop.xml Checked out external at revision 832876. ... As you can see the referred aop.xml file is imported from Common project through svn:externals % cd new-hdfs-workspace % ls -l src/test/aop/build aop.xml % ant injectfaults ... compile-fault-inject: [echo] Start weaving aspects in place [echo] Weaving of aspects is finished BUILD SUCCESSFUL Total time: 1 minute 11 seconds So everything seems to be working properly. Hope it helps to resolve the issue.
          Hide
          Eli Collins added a comment -

          Building at the previous change trunk@832540 works.

          Btw commenting out the include in build.xml results in:

          BUILD FAILED
          Target "injectfaults" does not exist in the project "Hadoop-Hdfs". It is used from target "jar-hdfs-test-fault-inject".

          Show
          Eli Collins added a comment - Building at the previous change trunk@832540 works. Btw commenting out the include in build.xml results in: BUILD FAILED Target "injectfaults" does not exist in the project "Hadoop-Hdfs". It is used from target "jar-hdfs-test-fault-inject".
          Hide
          Eli Collins added a comment -

          I'm getting a build failure running ant binary on a clean hdfs tree, this change makes build.xml reference test/aop/build/aop.xml but that file isn't checked in or generated first. Any ideas?

          BUILD FAILED
          /home/eli/src/apache-hadoop-hdfs/build.xml:262: Cannot find /home/eli/src/apache-hadoop-hdfs/src/test/aop/build/aop.xml imported from /home/eli/src/apache-hadoop-hdfs/build.xml

          Show
          Eli Collins added a comment - I'm getting a build failure running ant binary on a clean hdfs tree, this change makes build.xml reference test/aop/build/aop.xml but that file isn't checked in or generated first. Any ideas? BUILD FAILED /home/eli/src/apache-hadoop-hdfs/build.xml:262: Cannot find /home/eli/src/apache-hadoop-hdfs/src/test/aop/build/aop.xml imported from /home/eli/src/apache-hadoop-hdfs/build.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #97 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/97/)
          . Replace current fault injection implementation with one from. Contributed by Konstantin Boudnik

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #97 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/97/ ) . Replace current fault injection implementation with one from. Contributed by Konstantin Boudnik
          Hide
          Konstantin Boudnik added a comment -

          And this one doesn't need to be backported to 0.21 branch, for its implementation is Ok as it is - it should server the purpose.

          Show
          Konstantin Boudnik added a comment - And this one doesn't need to be backported to 0.21 branch, for its implementation is Ok as it is - it should server the purpose.
          Hide
          Konstantin Boudnik added a comment -

          Thanks for the review, Nicholas.
          I've just committed this.

          Show
          Konstantin Boudnik added a comment - Thanks for the review, Nicholas. I've just committed this.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1
          Tried the patch. It worked well.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 Tried the patch. It worked well.
          Hide
          Konstantin Boudnik added a comment -

          I have executed variety of FI related target with this patch and everything seems to be working. Also, I have created instrumented jar files before and after the patch's application. They are the same. So, the patch is good to go, it seems.

          Show
          Konstantin Boudnik added a comment - I have executed variety of FI related target with this patch and everything seems to be working. Also, I have created instrumented jar files before and after the patch's application. They are the same. So, the patch is good to go, it seems.
          Hide
          Konstantin Boudnik added a comment -

          Synch'ed with HADOOP-6347 implementation.

          Show
          Konstantin Boudnik added a comment - Synch'ed with HADOOP-6347 implementation.
          Hide
          Konstantin Boudnik added a comment -

          Synch'ed with trunk

          Show
          Konstantin Boudnik added a comment - Synch'ed with trunk
          Hide
          Konstantin Boudnik added a comment -

          Initial patch

          Show
          Konstantin Boudnik added a comment - Initial patch

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development