Details

    • Type: Umbrella Umbrella
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: test
    • Labels:
      None

      Description

      We're up to 2hrs 15mins running tests on jenkins which is kinda crazy. Akash started a thread over in dev on fixing this: http://search-hadoop.com/m/cZjDH1ykGIA/Running+UnitTests+before+submitting+a+patch&subj=Running+UnitTests+before+submitting+a+patch

      Jessie brings up http://maven.apache.org/plugins/maven-failsafe-plugin/ which looks like something we need to start using; having a tiering of our testing would make sense to me.

        Issue Links

          Activity

          Hide
          ramkrishna.s.vasudevan added a comment -

          +1 for this JIRA

          Show
          ramkrishna.s.vasudevan added a comment - +1 for this JIRA
          Hide
          Jesse Yates added a comment -

          Been thinking about the issue of how we want to handle breaking out the integration tests.

          Option 1
          we can have all the tests (integration and unit) in a single source folder (hbase/src/test/java/*) which then uses standard conventions for naming to ensure things are run correctly - ie at the right time

          • Unit tests would remain in the standard format ( **/*Test.java, **/Test*.java, **/*TestCase.java)
          • integration tests would move to a similar format ( **/*IT.java, **/IT*.java, **/*ITCase.java)

          we can move integration tests to their own sub-module of hbase. This would be a little more of a radical change.

          Show
          Jesse Yates added a comment - Been thinking about the issue of how we want to handle breaking out the integration tests. Option 1 we can have all the tests (integration and unit) in a single source folder (hbase/src/test/java/*) which then uses standard conventions for naming to ensure things are run correctly - ie at the right time Unit tests would remain in the standard format ( **/*Test.java, **/Test*.java, **/*TestCase.java) integration tests would move to a similar format ( **/*IT.java, **/IT*.java, **/*ITCase.java) we can move integration tests to their own sub-module of hbase. This would be a little more of a radical change.
          Hide
          Jesse Yates added a comment -

          (sorry, miss click - here are the rest of the details)

          Option 2
          We move refactor hbase into a top level pom w/ several sub-packages. This helps to break out different concerns. It will also be good later if we want to break out core into a couple different pieces. Right now this would just entail a slight change in structure:

          hbase/
             /hbase-core
             /hbase-it
          

          where hbase-it would have the integration tests, utils for the tests, and unit tests of those utils. It would be a 'full' package. Again we stick to the above naming convention (or specify our own) for integration tests

          Option 3
          Similar to the above, but we only make hbase-it a submodule of hbase (leave the structure the same). This still remains a full featured package. The directory structure looks like this:

          hbase/
            /src/main/java
            ...
            /hbase-it
              /src/main/java
              ...
          

          Right now, I can't figure out how to get the maven-failsafe-plugin to read from a directory other than /src/test/java, but if I can figure it out, that would be best when combined with Option 1 - keeps same structure, breaks out integration tests.

          Given that, I am more inclined to go with Option 2 or 3. These helps break out concerns into different packages, rather than munging them all together.

          Working on a patch now for #3 - its the simplest to break out. #2 wouldn't be terrible, but would require more delicacy. #1 is simple and is a matter of minutes, if we want to go that route.

          What does everyone think?

          Show
          Jesse Yates added a comment - (sorry, miss click - here are the rest of the details) Option 2 We move refactor hbase into a top level pom w/ several sub-packages. This helps to break out different concerns. It will also be good later if we want to break out core into a couple different pieces. Right now this would just entail a slight change in structure: hbase/ /hbase-core /hbase-it where hbase-it would have the integration tests, utils for the tests, and unit tests of those utils. It would be a 'full' package. Again we stick to the above naming convention (or specify our own) for integration tests Option 3 Similar to the above, but we only make hbase-it a submodule of hbase (leave the structure the same). This still remains a full featured package. The directory structure looks like this: hbase/ /src/main/java ... /hbase-it /src/main/java ... Right now, I can't figure out how to get the maven-failsafe-plugin to read from a directory other than /src/test/java, but if I can figure it out, that would be best when combined with Option 1 - keeps same structure, breaks out integration tests. Given that, I am more inclined to go with Option 2 or 3. These helps break out concerns into different packages, rather than munging them all together. Working on a patch now for #3 - its the simplest to break out. #2 wouldn't be terrible, but would require more delicacy. #1 is simple and is a matter of minutes, if we want to go that route. What does everyone think?
          Hide
          Todd Lipcon added a comment -

          My preference is to use JUnit category annotations - with SUREFIRE-329 we can easily separate long tests from short. Moving tests to separate subdirs seems a little painful to me...

          I agree that true integration/system tests should go to a separate dir - but those should be able to be run against a real cluster without ever accessing daemons directly. We have many minicluster tests that inject failures, etc, and to call them integration tests is a bit misleading. Rather they're just "long functional tests".

          Show
          Todd Lipcon added a comment - My preference is to use JUnit category annotations - with SUREFIRE-329 we can easily separate long tests from short. Moving tests to separate subdirs seems a little painful to me... I agree that true integration/system tests should go to a separate dir - but those should be able to be run against a real cluster without ever accessing daemons directly. We have many minicluster tests that inject failures, etc, and to call them integration tests is a bit misleading. Rather they're just "long functional tests".
          Hide
          Jesse Yates added a comment -

          I agree that those kind of tests still count as functional. When you are injecting faults into it, I would also worry about whats going to happen on the backend, when you are reusing the minicluster for multiple tests.

          It would be cool if we can then break out short and long running functional tests and run them as desired.

          However, as Doug commented on the thread though, the mini cluster is used in a lot of cases to do true functional testing (spin up mini cluster w/ a single node, do some stuff, then shut it down). I feel like there should be a fair number of non-daemon-interacting 'unit' tests which really shouldn't be considered unit tests.

          I don't have any experience tying in annotations to the build system, but I would be interested to see how that would all work.

          I feel like it still makes sense to break out integration tests, we just need to be careful about how we do it. What do you think?

          Show
          Jesse Yates added a comment - I agree that those kind of tests still count as functional. When you are injecting faults into it, I would also worry about whats going to happen on the backend, when you are reusing the minicluster for multiple tests. It would be cool if we can then break out short and long running functional tests and run them as desired. However, as Doug commented on the thread though, the mini cluster is used in a lot of cases to do true functional testing (spin up mini cluster w/ a single node, do some stuff, then shut it down). I feel like there should be a fair number of non-daemon-interacting 'unit' tests which really shouldn't be considered unit tests. I don't have any experience tying in annotations to the build system, but I would be interested to see how that would all work. I feel like it still makes sense to break out integration tests, we just need to be careful about how we do it. What do you think?
          Hide
          stack added a comment -

          Regards 'Option 2', the break-up of hbase into modules is going to happen in TRUNK for other reasons so you can assume submodules a non-obstacle – for TRUNK; modules will not happen for 0.92 branch.

          On integration tests, going by Todd definition, we need to do those too. On "long-running functional tests", yes would be grand to break them apart. I'd be fine with renaming long tests with *LongRunningFunctionalTest suffix or some such so we could distinguish them before a SUREFIRE-329 shows up (On SUREFIRE-329, Todd, you get around).

          Show
          stack added a comment - Regards 'Option 2', the break-up of hbase into modules is going to happen in TRUNK for other reasons so you can assume submodules a non-obstacle – for TRUNK; modules will not happen for 0.92 branch. On integration tests, going by Todd definition, we need to do those too. On "long-running functional tests", yes would be grand to break them apart. I'd be fine with renaming long tests with *LongRunningFunctionalTest suffix or some such so we could distinguish them before a SUREFIRE-329 shows up (On SUREFIRE-329, Todd, you get around).
          Hide
          Todd Lipcon added a comment -

          Highly recommend this article: http://googletesting.blogspot.com/2009/07/software-testing-categorization.html

          I'll keep campaigning to get SUREFIRE-329 in a release

          Show
          Todd Lipcon added a comment - Highly recommend this article: http://googletesting.blogspot.com/2009/07/software-testing-categorization.html I'll keep campaigning to get SUREFIRE-329 in a release
          Hide
          Jesse Yates added a comment -

          We could easily incorporate splitting out long run tests as part of the split recommended here.

          Show
          Jesse Yates added a comment - We could easily incorporate splitting out long run tests as part of the split recommended here.
          Hide
          Jesse Yates added a comment -

          And by "here" I mean HBASE-4336.

          Show
          Jesse Yates added a comment - And by "here" I mean HBASE-4336 .
          Hide
          Mikhail Bautin added a comment -

          If it is possible to annotate integration tests differently from unit tests without moving/renaming a lot of files, that would be a much better approach. We have a lot of patches queued up for open-sourcing, and moving a lot of files will make it very difficult to integrate them into trunk.

          Show
          Mikhail Bautin added a comment - If it is possible to annotate integration tests differently from unit tests without moving/renaming a lot of files, that would be a much better approach. We have a lot of patches queued up for open-sourcing, and moving a lot of files will make it very difficult to integrate them into trunk.
          Hide
          Jesse Yates added a comment -

          Right now its not too hard to get the integration tests named differently to run in a separate phase (basically add in the failsafe plugin and rename them all to */IT.java, or whatever regex we want). However with the split proposed in HBASE-4336, everything is going to move around anyways, so we might as well just get it done all together.

          Granted there is going to be some serious pain when we break out into modules.

          Adding ticket (HBASE-4454) so we can just do the integration tests with naming, for the moment.

          Show
          Jesse Yates added a comment - Right now its not too hard to get the integration tests named differently to run in a separate phase (basically add in the failsafe plugin and rename them all to * /IT .java, or whatever regex we want). However with the split proposed in HBASE-4336 , everything is going to move around anyways, so we might as well just get it done all together. Granted there is going to be some serious pain when we break out into modules. Adding ticket ( HBASE-4454 ) so we can just do the integration tests with naming, for the moment.

            People

            • Assignee:
              Unassigned
              Reporter:
              stack
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development