ITEST should be a derivative of BIGTOP_HOME. And the latter might have a decent default, don't you think? E.g. most likely the tests are run from the bigtop directory, so perhaps it makes sense to check if the current dir has the stuff that the typical BIGTOP_HOME would.

Hi david. I think its time to add itest source dependency as the top level dependency. that way we don't need to reference a version at all.

I added itest source dependency into the mapreduce smoke tests. I think we can easily bump that up to the top level gradle file, do you agree ? This snippet indeed implements cos's idea, but i should have done it at the top level (in build.gradle) instead of only for one specific test.

 40 
 41 sourceSets {
 42   main {
 43      groovy {
 44        srcDirs = [
 45          "$System.env.BIGTOP_HOME"+"/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/",
 46          "$System.env.BIGTOP_HOME"+"/bigtop-test-framework/src/test/groovy/org/apache/bigtop/itest/"
 47         ]
 48    }
 49   }

Then we can eliminate ITEST environmental variable entirely, and just use it from source, so that smoke-tests always has the up to date ITEST libraries. This solves a problem that has bit us in the past (BIOGTOP-1524).

• David Capwell : Shall we move the itest source inclusion to top level gradle, and eliminate ITEST var, for the direction of this patch ?
• Konstantin Boudnik does that answer your other question about it being a derivative of bigtop home ? Indeed, you were right, but i only implemented this for mapreduce/ tests, not glbally.

jay vyas makes sense.

Yup, it does. Thanks.

okay, this should allo us to run itest from source now

untested except a quick run that it compiles and tests can start.

i can look deeper this wk or anyone feel free to take it over

Having a hard time getting this to be at the top level gradle; not sure how to merge source sets between projects. Is there a reason that itest is managed outside of this gradle? Life will be easier if you just have one groovy and everything is a sub-project from there.

The simplest way to implement this would be to have bigtop-tests that has one root gradle project. Here itests would live, and so would smoke. Under smoke you would have the same setup, but they are just sub projects, so ':smoke-tests:mapreduce' vs ':mapreduce'.

thoughts?

I am not sure about this requirement to run iTest from the source? Why it can not be a dependency of the tests? After all, if one runs unit tests TestNG nor JUnit are used in the source code form... Am I missing something?

i believe that this is caused by the fact that itest is maven and the tests are gradle (aka not same parent build). If you depend on a version, you would first need to compile it (don't see anything wrong with that). Another thing, every release would need to update the version of itest to match what you are building. I don't believe gradle has a equivalent of mvn version plugin so you would have to update this at all call sites. You could define the version at the top level or in gradle properties, but you would still have to update.

If people don't want to move bigtop-test-framework to use the same build system, I could send a patch that does a require on the jar. This would work off your local m2 repo.

Konstantin Boudnik jay vyas thoughts on working with user's local m2 repo and keeping the version in gradle.properties?

I believe the overall aim is to move to Gradle and eventually eradicate maven, right? If so - would we be better of by wrapping existing Maven iTest module into Gradle so it becomes transparent for the Gradle parts of the system, like new smoke tests? This will allow to preserve the Maven parts of the tests in the working condition, along with their dependency on iTest. BTW, the way Maven'ized tests depend on iTest is by enforcing the iTest artifact re-installation into local .m2/ and then pulling it as the dependency.

Later we'll figure out how to completely replace maven bits with Gradle and that will be the end of it. But I am still very much dislike the idea of source dependency from tests to the framework.

1) Why run from source ?

Its a simple way to ensure that people using bigtop smoke tests :

• don't need to run mvn install or anything else ( i realize this can be abstacted into yet another wrapper, but for my response to that, see (2) below) .
• can modify Itest if they want, for some debugging/hacking and see results easily w/o getting confused why their changes arent absorbed.
• always have the latest fixes. Last updates to itest (BIGTOP-1524) broke the smoke-tests and made running latest itest a requirement for mapreduce smoke tests.

So, where in test-artifacts, we say "install maven, then build these jars, then you can run bigtop tests, oh and if you want to modify them - clean rebuild everything, and start over" in smoke-tests we say "clone bigtop - then you can run these tests. modifications? go for it. no change to your workflow.". Which framework would you prefer as a user? For me, the latter !

2) Should we wrap maven ? We played with that idea in BIGTOP-1195. The conclusion was : BEtter to build a easy to use framework with lower barrier to entry, then just wrap a hard to use framework in a pretty UI. So , BIGTOP-1222 was born - a new framework which was so easy to use, and direct at the same time - that no wrappers at all were needed...

Here are the benefits of the gradle smoke tests.

• Still have perfect JVM integration : We don't lose compilation against known sources.
• don't need to scan 3 different directories (test-framework,test-execution,test-artifacts) to determine the code path for their tests. this is a complete show-stopper for anyone who is not a maven guru.
• retain backward compatibliity w/ test-artifacts (i suggest we migrate them into the new framework but wont start that debate here).
• run way faster, and way easier, literally, I think 10x faster last time i checked. this means you can really rapidly test and run smoke-tests on a real cluster and modify them while developing.
• no roadblocks to people who arent' maven/gradle experts. no hidden dependencies.
• have some extra groovy tests? just drop em in a folder! they will run automatically.
• inherit the easy to use gradle interface gradlew compileGroovy test -Dsmoke.tests=pig
• will automatically run inside the bigtop auto-testing infrastructure which runs on virtualbox or docker in the click of a button

Now, if we swap out the elgant and direct gradle->compile classes->groovy test runtime w/ gradle->maven->jar->jar runner i am seeing 5 of those benefits gone (or severly comprimised). The original JIRA BIGTOP-1222, we decided that test-artifacts was only useful for the purpose of freezing versoins of jars in binary test-artifacts. for all other tests, smoke-tests approach works perfectly. So If we want to keep test-artifacts I promise to keep up the work to make sure that smoke-tests can use the source code from test-artifacts but i really hope we dont take smoke-tests, which can run on any java enabled linux machine in 1 minute, into something that had a bunch of dependencies, which took 8 minutes to run, and was impossible /error prone to modify at runtime. I ran into that alot w/ maven - because I'd need to modify something in, say, a pig test, then i'd need to clean the pig jar, and re run, hence all the work we put into BIGTOP-1195 and BIGTOP-1222 to come to a cleaner decision. Any QE team can easily adopt the current smoke-tests framework and add their own tests - not at all the case w/ the maven style of jar based tests.

P.S. The code inside of test-artifacts is awesome and super useful, im only advocating that we run them in a way which is common to integration testers : directly from the sources, not from some secret jar nested in 10 levels of .m2 directories, which takes 10 minutes to modify and re run.

As someone new to the project, finding how to run the tests is rather painful; and this has nothing to do with calling maven multiple times.

The big issue I am facing now is that everything that makes up bigtop testing is scattered all over the place. If this gets fixed then I could care less for gradle vs maven (could easily build mvnw).

Would people be fine with this jira really just restructuring the tests and getting them in one build system? I agree with Konstantin Boudnik that the source way is not the nicesest and as a new user very confusing.

Thoughts?

hmmm.. i may be in the minority on this one ... (David Capwell i definetly agree good to restructure all tests to a single system.)

if running from jar files is less confusing (not sure why running a jar file is easier than running from source), we can go back to using run 4 maven commands which take 8 minutes instead of one gradle command that takes 30 seconds.

but at the very least : lets not wrap maven into gradle (unless maven really is doing something we cant easily do in gradle)? that won't solve anything other than just making two levels of java indirection for something that really could just be done in a shell script.

but at the very least : lets not wrap maven into gradle. that won't solve anything other than just making two levels of java indirection for something that really could just be done in a shell script.

I agree, gradle calling maven makes no sense. My point was to do something like the following

$ ls -1 bigtop-tests
itest
smoke
regression
build.gradle
gradle
gradlew

So I can then do the following

./gradlew :smoke:test -Dsmoke.tests=mapreduce
./gradlew :regression:test

This would solve the itest problem since you are now in one build system, so for mapreduce, its build.gradle says

dependencies {
  compile project(':itest')
}

Gradle will only compile whats needed, so this will be just as fast as using source, if not faster since you no longer require to recompile itest on every run.

This also sets it up to solve my pain-point #2... why do I need gradle on my hadoop cluster in order to run bigtop? What I really want is a tarball (minus hadoop stack libs, those should be added to classpath from the host) and a config file that states what to run.

David, I actually agree with your last proposal - at least the way I decided to read it But seriously, I am not advocating to not run smokes from source code (as Jay may have alluded above); I am not either pushing for the wrappers - was simple exploration of the idea and thanks for reminding that the earlier attempt wasn't a success.

As for BIGTOP-1524 - I think this wasn't a well thought-through fix. I'd venture a guess that it was a quick way to address the bug introduced in iTest along with the failure injection mechanism, which was fixed later separately. However, I don't really see why in BIGTOP-1524 we couldn't depend on the iTest artifact and instead added the direct source code dependency? And not I think we are trying to provide a bigger fix to already suboptimal solution ie source code dependency from iTest.

I like David's proposal above - I really do. With one exception: let's keep the iTest to be the framework. And as such - let's not include it into the tests suites as in the above example. Because the minute will do it we'll have to either:

• keep all the tests in the same place, e.g. under bigtop-tests
• or have two copies of iTest source one of each for smokes and mavenized tests
  The source code dependency to a framework is a bad design - sorry if I hurt anyone's feeling. These things are orthogonal and should be separated.

Getting back to the mapreduce smoke: it declared dependency on junit but not on itest which is just yet another framework with extensions to junit, etc. Perhaps the problem we need to solve is how to publish/install iTest artifacts instead of simply bridging to its source? Say, I can get through mapreduce smoke compilation after applying this patch:

diff --git bigtop-tests/smoke-tests/mapreduce/build.gradle bigtop-tests/smoke-tests/mapreduce/build.gradle
index 83a16a6..c5bc5ab 100644
--- bigtop-tests/smoke-tests/mapreduce/build.gradle
+++ bigtop-tests/smoke-tests/mapreduce/build.gradle
@@ -20,12 +20,13 @@ apply plugin: 'groovy'
 
 repositories {
   mavenCentral()
+  mavenLocal()
 }
 
 dependencies {
   //needed to avoid groovy not on classpath error.
   testCompile module('org.codehaus.groovy:groovy:1.8.0')
-  compile group: 'junit', name: 'junit', version: '4.11', transitive: 'true'
+  compile group: 'org.apache.bigtop.itest', name: 'itest-common', version: '0.9.0-SNAPSHOT', transitive: 'false'
   testCompile group: 'org.apache.hadoop', name: 'hadoop-common', version: hadoopVersion, transitive: 'true'
   compile group: 'org.apache.hadoop', name: 'hadoop-hdfs', version: hadoopVersion, transitive: 'true'
   compile "org.codehaus.groovy:groovy:1.8.0"
@@ -39,14 +40,6 @@ def tests_to_include() {
 }
 
 sourceSets {
-  main {
-     groovy {
-       srcDirs = [
-         "$System.env.BIGTOP_HOME"+"/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/",
-         "$System.env.BIGTOP_HOME"+"/bigtop-test-framework/src/test/groovy/org/apache/bigtop/itest/"
-       ]
-   }
-  }
   test {
     resources {
       srcDirs = [
@@ -57,8 +50,6 @@ sourceSets {
     groovy {
       srcDirs = [
         "$System.env.BIGTOP_HOME"+"/bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/mapreduce",
-        "$System.env.BIGTOP_HOME"+"/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/",
-        "$System.env.BIGTOP_HOME"+"/bigtop-test-framework/src/test/groovy/org/apache/bigtop/itest/"
       ]
       exclude {
         FileTreeElement elem -> (doExclude(elem.getName()))

BTW, looks like smoke-tests README needs to be fixed as it doesn't clearly states that smokes should be run from the smoke test directory and not from the top-level one.

And yeah - I am totally Ok if instead of patching on top patching will workout a better testing system that will incorporate the needs of smoke tests and bigger cluster tests.

1) im an agreement also w/ davids idea. if you are suggesting that gradle transparently compiles itest into a jar , thats good. and i really like a unified interface. and am not married to any implementation per se.

2) Konstantin Boudnik your right - as long as we can use itest without depending on an old version, its ok.

3) Can someone here please tell me, what the advantage of compiling the pig smoke test, which is just a groovy file, into a jar file is before running it ? it takes longer, and is less easy to modify. im totally against making jars for ecosystem tests. it totally alienates the entire non-java community. i've never once run a suite of tests on the hadoop ecosystem that i didnt want to modify or play with something.

Am i crazy ? Or is there some huge benefit to hiding shell commands inside a jar that im missing?

You are making a good point Jay - running tests from the source code, e.g. in a script form is very convenient. We also need to have a way to publish test jars along with the released software stack, because the premise and the value-add of Bigtop was and is to guarantee that the software stack is validated with fist-class tests built for that particular stack. You can't do it with just source code - you have to have dependency and version management for that.

That said: for dev. purposes - running from source code is great. But we need to have a way to publish the jars as a part of the release. I believe Gradle should allow us to do both. Am I making sense?

yup makes sense. ithink were all in agreement. and i think when i said from source i meant what david was saying in the compile project(':itest'). i almost think we are all in 100% agreement.

I guess we are, as far as the test framework - be it junit or itest - isn't a part of the test source, but is an external dependency

ok. agreed to make davids proposal a requirement for the new integrated framework ?

$ ls -1 bigtop-tests
itest
smoke
regression
build.gradle
gradle
gradlew

Guys, please correct me if I am wrong, but how moving itest source tree into test tree address the separation concern I've expressed earlier?

Also, what would constitutes a regression suite?

1) your right cos, IMO for itest, keeping in bigtop-test-framework is okay .
2) i figured regression is a catch all for the tests that we have that smoke tests. like fine grained tests, for example hadoop cli, pig-hbase integration, and so on. could have a better name i guess.

David Capwell Konstantin Boudnik Roman Shaposhnik lets try to chat about this sometime this wk, if we are all free, and nail down a good architecture for this guy/

Please feel free to setup a call - I should be able to flex my schedule to meet it.

Heading to Tahoe friday, so any other day I can make myself available.

Before then, I do have one question: what is the advantage of not putting itest in the same build system as the tests? Did a grep over the source, and it looks like only the bigtop-tests directory uses it.

jay vyas correct, I just placed the word regression since im not sure what the other tests are. When I look at test artifacts dir, looks like most are smoke tests.

what is the advantage of not putting itest in the same build system as the tests

the advantage is perhaps of pure design aesthetic. iTest is a framework, that has its own unit and integration tests and can be used outside of the Bigtop, e.g. it provides good primitives for order test runs, etc. Hence is me pressing on this as it provides functionality for the test and should be separated in a different module. I never said it shouldn't be a part of the same build system. In fact it should be: either of Gradle or Maven. Doesn't have to be in both.

Thanks for the clarification. Think we are all on the same page then.

UPDATE

1) in order to unify, we need to make itest build via gradle, and then call that gradle task from smoke tests. so im working that angle now.
2) after that we need to migrate all the groovy files from test-artifacts into a (hopefully less nested and simpler) directory that is under a common subdir w/ smoke test tests.

ive just created the jira for the fist subtask: Gradlizing itest. after that, we can proceed with this jira ! quick patch attached in BIGTOP-1621

One issue with the last point here: I'd love to preserve an ability to deploy test-artifacts as in the current incarnation. I've explained the reason elsewhere, let me repeat it again: a particular state of the test code comprises what we used to call "test stack". It reflects the state of the software stack, defined in the BOM. Having an ability to build the stack and test it with the readily available set of tests - not just scripts/source code, but first class artifacts - seems like a huge value-add to me.

Shall we move it to 1.1.0 while we are figuring out the best approach in light of BIGTOP-1621? What do you think, David Capwell?

Makes sense to me.

BTW removing my name from here since I won't be able to work in this

Guys, I want to go ahead and commit this patch, because stepping on silently missing BIGTOP_HOME is a pain. Whatever has been discussed here is still valid of course, but we should be able to work on that in a subsequent JIRAs, instead of dragging the real fix out forever.

The patch will be committed in BIGTOP-2175.

The patch is moved to a different JIRA. This ticket is repurposed for a wider scope.

We are literally one tiny patch away from completing this!

Instead of going into backlog it needs to be resolved as all the subtasks are done. Hence, the umbrella ticket is done as well. Resolving

Thanks !!!