ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-725

Test organization, methodology, and infrastructure improvements

    Details

    • Type: Test Test
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: build, tests
    • Labels:
      None

      Description

      This is an umbrella feature for a number of improvements needed to happen in ZK test realm

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment -

          Cos, what's best practice for passing information into a test? In this case say we are passing the server port information into the client tests. How best to do this? jvm parameters (-Dzk.server.host1=host1:1234)? parameterized junit tests seem overkill for just a single test, no?

          Unfortunately, JUnit (at least currently) isn't very flexible in this regard.

          TestNG is a way more advanced in this regard, but their sameVM model of test execution makes test development a more cumbersome task. However, the tests are coming out are more elegant and correct in comparison with JUnit approach of killing a VM after a test case execution. We already had an exercise with TestNG and you know what I meant

          Possible ways to parametrize tests in JUnit are:

          • sys. props in the command line/runtime as you have pointed out above
          • sys. props set in a properties file
          • using Fixtures aka @Before and @After methods. Using fixtures one can develop an elaborate setup method which might set certain states before a test case is started. The data for setting such state might be read from a file containing a set of parameters per line. Or it might be more "sophisticated" XML file. Or a configuration factory. Or something else...

          Apparently, first two approaches are inferior to the latter way. However, it requires more ground work. Benefits are numerous. Not the least one is a significant simplification of build process.

          Show
          Konstantin Boudnik added a comment - Cos, what's best practice for passing information into a test? In this case say we are passing the server port information into the client tests. How best to do this? jvm parameters (-Dzk.server.host1=host1:1234)? parameterized junit tests seem overkill for just a single test, no? Unfortunately, JUnit (at least currently) isn't very flexible in this regard. TestNG is a way more advanced in this regard, but their sameVM model of test execution makes test development a more cumbersome task. However, the tests are coming out are more elegant and correct in comparison with JUnit approach of killing a VM after a test case execution. We already had an exercise with TestNG and you know what I meant Possible ways to parametrize tests in JUnit are: sys. props in the command line/runtime as you have pointed out above sys. props set in a properties file using Fixtures aka @Before and @After methods. Using fixtures one can develop an elaborate setup method which might set certain states before a test case is started. The data for setting such state might be read from a file containing a set of parameters per line. Or it might be more "sophisticated" XML file. Or a configuration factory. Or something else... Apparently, first two approaches are inferior to the latter way. However, it requires more ground work. Benefits are numerous. Not the least one is a significant simplification of build process.
          Hide
          Patrick Hunt added a comment -

          It seems to be that client/server separation and build re-org seem like a logical first steps. Mockito + unit tests might come next. Stress/performance are ok to be added lately, because these might require some additional infrastructure/framework efforts.

          seems reasonable. Henry, I have a pending test patch, now that 3.3.0 is in the bag can you review/commit that to trunk?

          Cos, what's best practice for passing information into a test? In this case say we are passing the server port information into the client tests. How best to do this? jvm parameters (-Dzk.server.host1=host1:1234)? parameterized junit tests seem overkill for just a single test, no?

          Show
          Patrick Hunt added a comment - It seems to be that client/server separation and build re-org seem like a logical first steps. Mockito + unit tests might come next. Stress/performance are ok to be added lately, because these might require some additional infrastructure/framework efforts. seems reasonable. Henry, I have a pending test patch, now that 3.3.0 is in the bag can you review/commit that to trunk? Cos, what's best practice for passing information into a test? In this case say we are passing the server port information into the client tests. How best to do this? jvm parameters (-Dzk.server.host1=host1:1234)? parameterized junit tests seem overkill for just a single test, no?
          Hide
          Konstantin Boudnik added a comment -

          re 1) we should fix the test pkging, also split out units vs func etc... JUnit4 has "category" annotations. We should use these to define things like long/short running tests, etc... Not sure if we want to separate diff test types into diff directories (unit/func/perf/etc...) or use cats, or some combination but we need to think about that as well. src/java/test vs say src/java/units & src/java/func etc... Konstantin ideas for best practices here?

          My recent experience with having a physical taxonomy of tests (unit/func/perf) in Hadoop seems to be pretty positive. I.e. it makes build layout simpler. Tests categorization is more clear this way, too. JUnit Categories are rather primitive in compare with Spring or TestNG ones (or at least it used to be like that last time I've checked).

          re 4) maybe create a jira for it and link? I'd like to see perf tests run as part of some sort of "nightly" build. Not sure if apache hudson can do this or not though... Konstantin any suggestions on this? What does MR/HDFS do for this?

          For a project with a relatively small test base it might be possible to execute everything every, say, night. For MR/HDFS we don't run any performance tests on Apache infrastructure. However, whatever JUnit based tests we have are all ran a couple times a day.

          What is important to mention, is that performance/stress results need to be properly trended/analyzed. Otherwise, it doesn't make much sense to bother with these types of tests.

          Having subtasked JIRAs for stress and performance tests are good idea, because these are pretty different and are better be tracked separately.

          5) there's been some interest to move to maven, would this help/hurt us and should we do that beore/after doing this? is there a "best" order for doing these changes?

          It seems to be that client/server separation and build re-org seem like a logical first steps. Mockito + unit tests might come next. Stress/performance are ok to be added lately, because these might require some additional infrastructure/framework efforts.

          Show
          Konstantin Boudnik added a comment - re 1) we should fix the test pkging, also split out units vs func etc... JUnit4 has "category" annotations. We should use these to define things like long/short running tests, etc... Not sure if we want to separate diff test types into diff directories (unit/func/perf/etc...) or use cats, or some combination but we need to think about that as well. src/java/test vs say src/java/units & src/java/func etc... Konstantin ideas for best practices here? My recent experience with having a physical taxonomy of tests (unit/func/perf) in Hadoop seems to be pretty positive. I.e. it makes build layout simpler. Tests categorization is more clear this way, too. JUnit Categories are rather primitive in compare with Spring or TestNG ones (or at least it used to be like that last time I've checked). re 4) maybe create a jira for it and link? I'd like to see perf tests run as part of some sort of "nightly" build. Not sure if apache hudson can do this or not though... Konstantin any suggestions on this? What does MR/HDFS do for this? For a project with a relatively small test base it might be possible to execute everything every, say, night. For MR/HDFS we don't run any performance tests on Apache infrastructure. However, whatever JUnit based tests we have are all ran a couple times a day. What is important to mention, is that performance/stress results need to be properly trended/analyzed. Otherwise, it doesn't make much sense to bother with these types of tests. Having subtasked JIRAs for stress and performance tests are good idea, because these are pretty different and are better be tracked separately. 5) there's been some interest to move to maven, would this help/hurt us and should we do that beore/after doing this? is there a "best" order for doing these changes? It seems to be that client/server separation and build re-org seem like a logical first steps. Mockito + unit tests might come next. Stress/performance are ok to be added lately, because these might require some additional infrastructure/framework efforts.
          Hide
          Patrick Hunt added a comment -

          Henry good questions. I believe the idea here is to pull together the efforts (whether subtasks or links) and coordinate them.

          re 1) we should fix the test pkging, also split out units vs func etc... JUnit4 has "category" annotations. We should use these to define things like long/short running tests, etc... Not sure if we want to separate diff test types into diff directories (unit/func/perf/etc...) or use cats, or some combination but we need to think about that as well. src/java/test vs say src/java/units & src/java/func etc... Konstantin ideas for best practices here?

          re 3) my expectation is we should use mockito wherever it's useful - in particular it seems like server side is high value. client is pretty thin, not sure how much can be mocked... We don't provide mocks of the ZooKeeper class itself, seems like tis would be useful for our users (test error conditions in particular). Regardless of where we use mockito we suffer from a big lack of "design for testability", so not only will this require big changes to the tests but also to the code itself. (also keep in mind we are working to integrate netty/avro which will each have big impact on current design)

          re 4) maybe create a jira for it and link? I'd like to see perf tests run as part of some sort of "nightly" build. Not sure if apache hudson can do this or not though... Konstantin any suggestions on this? What does MR/HDFS do for this?

          5) there's been some interest to move to maven, would this help/hurt us and should we do that beore/after doing this? is there a "best" order for doing these changes?

          seems like we need to do incremental, and have at least 1 contributor and 1 committer dedicated to this effort (or 2 commiters - so one can make the changes and the other can review/commit them, perhaps in parallel). I would be willing to dedicate a chunk of my available time to this effort.

          Show
          Patrick Hunt added a comment - Henry good questions. I believe the idea here is to pull together the efforts (whether subtasks or links) and coordinate them. re 1) we should fix the test pkging, also split out units vs func etc... JUnit4 has "category" annotations. We should use these to define things like long/short running tests, etc... Not sure if we want to separate diff test types into diff directories (unit/func/perf/etc...) or use cats, or some combination but we need to think about that as well. src/java/test vs say src/java/units & src/java/func etc... Konstantin ideas for best practices here? re 3) my expectation is we should use mockito wherever it's useful - in particular it seems like server side is high value. client is pretty thin, not sure how much can be mocked... We don't provide mocks of the ZooKeeper class itself, seems like tis would be useful for our users (test error conditions in particular). Regardless of where we use mockito we suffer from a big lack of "design for testability", so not only will this require big changes to the tests but also to the code itself. (also keep in mind we are working to integrate netty/avro which will each have big impact on current design) re 4) maybe create a jira for it and link? I'd like to see perf tests run as part of some sort of "nightly" build. Not sure if apache hudson can do this or not though... Konstantin any suggestions on this? What does MR/HDFS do for this? 5) there's been some interest to move to maven, would this help/hurt us and should we do that beore/after doing this? is there a "best" order for doing these changes? seems like we need to do incremental, and have at least 1 contributor and 1 committer dedicated to this effort (or 2 commiters - so one can make the changes and the other can review/commit them, perhaps in parallel). I would be willing to dedicate a chunk of my available time to this effort.
          Hide
          Henry Robinson added a comment -

          These are all excellent ideas - thanks for getting the ball rolling!

          Some general questions to spur discussion:

          1. There are a lot of tests in ZooKeeper, and it seems we'll have to rewrite most of them as part of this JIRA. Will it make sense to stage the rewrite process, maybe package by package or even finer grained? In general there's a dependency problem to be solved here, it would be good to figure out the details up front.
          2. How will we validate the new tests?
          3. One particular facility that we sorely need is the ability to deterministically execute certain interleavings of concurrent processes (e.g. two leader election protocols) - this would help us perform validation of protocols to their specification. It seems Mockito would also help us here - is it sensible to expand its use beyond the client API?
          4. I would also like to see a suite of stress tests built against each major component - is that beyond the scope of the JIRA?

          Thanks,
          Henry

          Show
          Henry Robinson added a comment - These are all excellent ideas - thanks for getting the ball rolling! Some general questions to spur discussion: 1. There are a lot of tests in ZooKeeper, and it seems we'll have to rewrite most of them as part of this JIRA. Will it make sense to stage the rewrite process, maybe package by package or even finer grained? In general there's a dependency problem to be solved here, it would be good to figure out the details up front. 2. How will we validate the new tests? 3. One particular facility that we sorely need is the ability to deterministically execute certain interleavings of concurrent processes (e.g. two leader election protocols) - this would help us perform validation of protocols to their specification. It seems Mockito would also help us here - is it sensible to expand its use beyond the client API? 4. I would also like to see a suite of stress tests built against each major component - is that beyond the scope of the JIRA? Thanks, Henry
          Hide
          Konstantin Boudnik added a comment -

          Here's the list of required improvements (particular sub-tasks will be added later):

          • logical and physical separation of client and server tests so client testing can be executed against an arbitrary server version
          • parametrization of client tests
          • adding Mockito framework to allow true unit testing (TUT) of client API
          • improvements in the testing logs to facilitate diagnostics
          • necessary build reorganization to enable all above.
          Show
          Konstantin Boudnik added a comment - Here's the list of required improvements (particular sub-tasks will be added later): logical and physical separation of client and server tests so client testing can be executed against an arbitrary server version parametrization of client tests adding Mockito framework to allow true unit testing (TUT) of client API improvements in the testing logs to facilitate diagnostics necessary build reorganization to enable all above.

            People

            • Assignee:
              Unassigned
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development