Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2261

AOP unit tests are not getting compiled or run

    Details

      Description

      The tests in src/test/aop are not getting compiled or run.

      1. hdfs-2261.patch
        146 kB
        Karthik Kambatla

        Issue Links

          Activity

          Hide
          Karthik Kambatla (Inactive) added a comment -

          Just checking again if we have a decision here. I think we should remove these tests or have a concrete plan on enabling them. Otherwise, one call always pull them in from previous branches.

          Show
          Karthik Kambatla (Inactive) added a comment - Just checking again if we have a decision here. I think we should remove these tests or have a concrete plan on enabling them. Otherwise, one call always pull them in from previous branches.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Decoupled the aspect dependency issue - created HADOOP-9550 to address that while we decide on a plan for the injection tests.

          Show
          Karthik Kambatla (Inactive) added a comment - Decoupled the aspect dependency issue - created HADOOP-9550 to address that while we decide on a plan for the injection tests.
          Hide
          Alejandro Abdelnur added a comment -

          Nicholas, since we mavenized hadoop this has not been build/tested. And a JIRA to wire it has been seating idle for almost 2 years, HADOOP-7481. Are you taking up finishing this work now? Else, I'd suggest deleting trunk; and whenever there is renewed interested dig it from SVN.

          Show
          Alejandro Abdelnur added a comment - Nicholas, since we mavenized hadoop this has not been build/tested. And a JIRA to wire it has been seating idle for almost 2 years, HADOOP-7481 . Are you taking up finishing this work now? Else, I'd suggest deleting trunk; and whenever there is renewed interested dig it from SVN.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          For other un-mavenized artifacts such as forrest doc, we fix them instead of removing them. Why we must remove the fault injection tests?

          > ... there is no way to compile/run them. I see that we were using ant the last time these were checked, and there was a way to run them.

          I don't agree. It is just that our maven is not yet configured to support them. I beg it is easy to add a maven ant task to run them.

          > ... Otherwise, we are essentially shipping releases with dead code and unused dependencies.

          For the AspectJ dependency, I am fine to remove it for the moment. Ideally, the main code and the other test codes should not depend on AspectJ, only the fault injection tests should.

          Show
          Tsz Wo Nicholas Sze added a comment - For other un-mavenized artifacts such as forrest doc, we fix them instead of removing them. Why we must remove the fault injection tests? > ... there is no way to compile/run them. I see that we were using ant the last time these were checked, and there was a way to run them. I don't agree. It is just that our maven is not yet configured to support them. I beg it is easy to add a maven ant task to run them. > ... Otherwise, we are essentially shipping releases with dead code and unused dependencies. For the AspectJ dependency, I am fine to remove it for the moment. Ideally, the main code and the other test codes should not depend on AspectJ, only the fault injection tests should.
          Hide
          Alejandro Abdelnur added a comment -

          and maven does not compile them either.

          +1 from my side. Nicholas, are you ok with this?

          Show
          Alejandro Abdelnur added a comment - and maven does not compile them either. +1 from my side. Nicholas, are you ok with this?
          Hide
          Karthik Kambatla (Inactive) added a comment -

          The patch removes tests, QA reports no tests because it doesn't understand the tests under aop/.

          Show
          Karthik Kambatla (Inactive) added a comment - The patch removes tests, QA reports no tests because it doesn't understand the tests under aop/.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Hi Nicholas,

          Thanks for your feedback now and earlier too. These tests might be valuable, but unfortunately there is no way to compile/run them. I see that we were using ant the last time these were checked, and there was a way to run them. Given that they aren't made to work with maven (close to 2 years), I think we should remove them from trunk (at least branch-2) now, and may be create another JIRA to revive them from previously working branches. Otherwise, we are essentially shipping releases with dead code and unused dependencies.

          What do you think?

          Show
          Karthik Kambatla (Inactive) added a comment - Hi Nicholas, Thanks for your feedback now and earlier too. These tests might be valuable, but unfortunately there is no way to compile/run them. I see that we were using ant the last time these were checked, and there was a way to run them. Given that they aren't made to work with maven (close to 2 years), I think we should remove them from trunk (at least branch-2) now, and may be create another JIRA to revive them from previously working branches. Otherwise, we are essentially shipping releases with dead code and unused dependencies. What do you think?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12582010/hdfs-2261.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-client hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4361//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4361//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582010/hdfs-2261.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-client hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4361//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4361//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Please don't remove hadoop-hdfs-project/hadoop-hdfs/src/test/aop. As checked previously, these tests were working fine.

          Show
          Tsz Wo Nicholas Sze added a comment - Please don't remove hadoop-hdfs-project/hadoop-hdfs/src/test/aop. As checked previously , these tests were working fine.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          A straight-forward patch that nukes the tests under hadoop-hdfs-project/hadoop-hdfs/src/test/aop/ and the associated dependencies (aspectj)

          Show
          Karthik Kambatla (Inactive) added a comment - A straight-forward patch that nukes the tests under hadoop-hdfs-project/hadoop-hdfs/src/test/aop/ and the associated dependencies (aspectj)
          Hide
          Karthik Kambatla (Inactive) added a comment -

          The tests under hadoop-hdfs-project/hadoop-hdfs/src/test/aop .

          Show
          Karthik Kambatla (Inactive) added a comment - The tests under hadoop-hdfs-project/hadoop-hdfs/src/test/aop .
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Just checking in. Looks like the tests are not being actively used, haven't been compiling for a long while now. I propose removing them ...

          Which tests are you suggesting to remove? Can you be more specific?

          Show
          Tsz Wo Nicholas Sze added a comment - > Just checking in. Looks like the tests are not being actively used, haven't been compiling for a long while now. I propose removing them ... Which tests are you suggesting to remove? Can you be more specific?
          Hide
          Alejandro Abdelnur added a comment -

          +1 for removing them.

          Show
          Alejandro Abdelnur added a comment - +1 for removing them.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Just checking in. Looks like the tests are not being actively used, haven't been compiling for a long while now. I propose removing them and the associated dependencies. Thoughts?

          Show
          Karthik Kambatla (Inactive) added a comment - Just checking in. Looks like the tests are not being actively used, haven't been compiling for a long while now. I propose removing them and the associated dependencies. Thoughts?
          Hide
          Eli Collins added a comment -

          The system tests were removed in HADOOP-8450, re-purposing this jira to get the aop tests compiling and running, looks like they're completely unhooked from the mvn build.

          Show
          Eli Collins added a comment - The system tests were removed in HADOOP-8450 , re-purposing this jira to get the aop tests compiling and running, looks like they're completely unhooked from the mvn build.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I would like to keep the aop unit tests. These tests did catch quite a few bugs in the past.

          Show
          Tsz Wo Nicholas Sze added a comment - I would like to keep the aop unit tests. These tests did catch quite a few bugs in the past.
          Hide
          Eli Collins added a comment -

          I'd be fine removing both, I've never seen them catch a bug and maintaining the infrastructure for them is a pain. New jira?

          Show
          Eli Collins added a comment - I'd be fine removing both, I've never seen them catch a bug and maintaining the infrastructure for them is a pain. New jira?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > In the meantime, should we disable the fault injection tests in HDFS?

          I am not sure if you aware that there are at least two types of fault injection tests.

          • aop unit tests (src/test/aop)
          • aop system tests (src/test/system)

          Interestingly, aop unit tests work fine. I just have tried "ant run-test-hdfs-fault-inject" and all tests are passed. However, aop system tests cannot be compiled as described in this JIRA (i.e. "ant test-system" fails). I might be wrong but it seems that no one is using aop system tests. If it is the case, how about we remove aop system tests?

          Show
          Tsz Wo Nicholas Sze added a comment - > In the meantime, should we disable the fault injection tests in HDFS? I am not sure if you aware that there are at least two types of fault injection tests. aop unit tests (src/test/aop) aop system tests (src/test/system) Interestingly, aop unit tests work fine. I just have tried "ant run-test-hdfs-fault-inject" and all tests are passed. However, aop system tests cannot be compiled as described in this JIRA (i.e. "ant test-system" fails). I might be wrong but it seems that no one is using aop system tests. If it is the case, how about we remove aop system tests?
          Hide
          Eli Collins added a comment -

          One thing we could for the the fault injection is to move all of it to a hadoop-faultinjection module, by doing that we won't have to publish the instrumented artifacts.

          +1

          In the meantime, should we disable the fault injection tests in HDFS?

          +1

          Show
          Eli Collins added a comment - One thing we could for the the fault injection is to move all of it to a hadoop-faultinjection module, by doing that we won't have to publish the instrumented artifacts. +1 In the meantime, should we disable the fault injection tests in HDFS? +1
          Hide
          Tom White added a comment -

          > One thing we could for the the fault injection is to move all of it to a hadoop-faultinjection module, by doing that we won't have to publish the instrumented artifacts.

          +1 that would make it more self-contained and hopefully easier to maintain.

          In the meantime, should we disable the fault injection tests in HDFS?

          Show
          Tom White added a comment - > One thing we could for the the fault injection is to move all of it to a hadoop-faultinjection module, by doing that we won't have to publish the instrumented artifacts. +1 that would make it more self-contained and hopefully easier to maintain. In the meantime, should we disable the fault injection tests in HDFS?
          Hide
          Alejandro Abdelnur added a comment -

          As mentioned in the dev alias:


          AOP stuff is now wired yet in common Mavenization, the instrumented JAR is not being created/deployed.

          From some of the messages, it seems related to that.

          HDFS Mavenization (HDFS-2096), which has been +1, does not attempt to run AOP stuff either, thus it would 'fix' this build failure for now. Later, when AOP is wired to Mavenization the fault injection test would be back to the build.

          At the moment, per Arun's request, we are holding on HDFS-2096 until MAPREDUCE-279 goes in.


          One thing we could for the the fault injection is to move all of it to a hadoop-faultinjection module, by doing that we won't have to publish the instrumented artifacts.

          Show
          Alejandro Abdelnur added a comment - As mentioned in the dev alias: AOP stuff is now wired yet in common Mavenization, the instrumented JAR is not being created/deployed. From some of the messages, it seems related to that. HDFS Mavenization ( HDFS-2096 ), which has been +1, does not attempt to run AOP stuff either, thus it would 'fix' this build failure for now. Later, when AOP is wired to Mavenization the fault injection test would be back to the build. At the moment, per Arun's request, we are holding on HDFS-2096 until MAPREDUCE-279 goes in. One thing we could for the the fault injection is to move all of it to a hadoop-faultinjection module, by doing that we won't have to publish the instrumented artifacts.
          Hide
          Eli Collins added a comment -

          Weird, even if I blow away my local m2 repo, check out a fresh hadoop-common tree, and just run ant -Dresolver=internal run-test-hdfs-fault-inject from hdfs it works for me. There is definitely a stale common jar somewhere in your build.

          Show
          Eli Collins added a comment - Weird, even if I blow away my local m2 repo, check out a fresh hadoop-common tree, and just run ant -Dresolver=internal run-test-hdfs-fault-inject from hdfs it works for me. There is definitely a stale common jar somewhere in your build.
          Hide
          Giridharan Kesavan added a comment -

          hdfs is resolving the latest hadoop common snapshot jars from the apache snapshot repo.
          verified this with the apache repo as well.

          https://repository.apache.org/content/groups/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT
          latest version of snapshot jars available in the snapshot repo:
          hadoop-common-0.23.0-20110815.215733-266-tests.jar
          hadoop-common-0.23.0-20110815.215733-266.jar
          
          console output of ivy which resolved the same snapshot version from apache repo.
          https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Commit/837/console
          
          [ivy:resolve] .. (0kB)
          
          [ivy:resolve]   [SUCCESSFUL ] org.apache.hadoop#avro;1.3.2!avro.jar (1011ms)
          [ivy:resolve] downloading
          https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-20110815.215733-266.jar
          ...
          
          [ivy:resolve] ........................................................................................................................................................................................................................
          (1667kB)
          [ivy:resolve] .. (0kB)
          [ivy:resolve]   [SUCCESSFUL ]
          org.apache.hadoop#hadoop-common;0.23.0-SNAPSHOT!hadoop-common.jar
          (1549ms)
          
          ivy-retrieve-common:
          [ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use
          'ivy.settings.file' instead
          [ivy:cachepath] :: loading settings :: file =
          /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Commit/trunk/ivy/ivysettings.xml
          
          ivy-resolve-hdfs:
          
          ivy-retrieve-hdfs:
          
          ivy-resolve-test:
          
          [ivy:resolve] downloading
          https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-20110815.215733-266-tests.jar
          
          Show
          Giridharan Kesavan added a comment - hdfs is resolving the latest hadoop common snapshot jars from the apache snapshot repo. verified this with the apache repo as well. https://repository.apache.org/content/groups/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT latest version of snapshot jars available in the snapshot repo: hadoop-common-0.23.0-20110815.215733-266-tests.jar hadoop-common-0.23.0-20110815.215733-266.jar console output of ivy which resolved the same snapshot version from apache repo. https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Commit/837/console [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.apache.hadoop#avro;1.3.2!avro.jar (1011ms) [ivy:resolve] downloading https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-20110815.215733-266.jar ... [ivy:resolve] ........................................................................................................................................................................................................................ (1667kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.23.0-SNAPSHOT!hadoop-common.jar (1549ms) ivy-retrieve-common: [ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead [ivy:cachepath] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Commit/trunk/ivy/ivysettings.xml ivy-resolve-hdfs: ivy-retrieve-hdfs: ivy-resolve-test: [ivy:resolve] downloading https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-20110815.215733-266-tests.jar
          Hide
          Eli Collins added a comment -

          This passes for me locally. Looks like on Jenkins it's not pickup up the latest common jars from maven.

          Show
          Eli Collins added a comment - This passes for me locally. Looks like on Jenkins it's not pickup up the latest common jars from maven.

            People

            • Assignee:
              Unassigned
              Reporter:
              Giridharan Kesavan
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:

                Development