Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1222

Simplify and gradleize a subset of the bigtop smokes

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.7.0
    • 0.8.0
    • build, tests
    • None

    Description

      (Rewritten the description for clarity)

      We need an easier way to run bigtop smoke tests, and gradle provides this:

      1) Easy to script/modify
      2) Human readable
      3) equally oriented towards both groovy and plain old java

      The advantage of this method to running smokes :

      1) No need to compile a jar : this is a costly step and not much value added, also creates indirection which can make debugging a broken test very hard.

      2) Simple: A smoke test doesnt need to make low level API calls or be compiled against the right APIs - rather, it should test the end user interface ("hive -q ....", "pig -x ....", "hadoop jar ....", and so on).

      3) Customizable: The smoke tests shouldnt require users to have to write XML and debug environmental variables / grep around for System properties etc. Rather, a high level controller should do all that checking for you.

      The initial idea was to write a python/bash implementation wrapper of scripts, but that was replaced by the idea of using gradle. The advantage of gradle is that we don't need to manually set the classpath and run groovy commands: Gradle wraps groovy scripts in their native java context quite nicely - but it doesnt add any other unnecessary overhead (xml, jar files, no need for complex xml tag wrappers for simple tasks - just plain groovy code).

      So, here the goal is just to create a nice, clean, extensible non-jar, non-API dependent gradle runner for the smoke tests which exersizes the hadoop cluster the same way a typical end-user would.

      Attachments

        1. newsmokes.png
          40 kB
          jay vyas
        2. BIGTOP-1222-2.patch
          23 kB
          jay vyas
        3. BIGTOP-1222.patch
          17 kB
          jay vyas
        4. BIGTOP-1222.patch
          17 kB
          jay vyas
        5. BIGTOP-1222.patch
          17 kB
          jay vyas
        6. BIGTOP-1222.patch
          26 kB
          jay vyas
        7. BIGTOP-1222.patch
          35 kB
          jay vyas
        8. BIGTOP-1222.patch
          5 kB
          Dawson Choong
        9. BIGTOP-1222.patch
          49 kB
          jay vyas
        10. BIGTOP-1222.patch
          64 kB
          jay vyas
        11. BIGTOP-1222.patch
          64 kB
          jay vyas
        12. BIGTOP-1222.patch
          47 kB
          jay vyas
        13. BIGTOP-1222.patch
          47 kB
          jay vyas
        14. BIGTOP-1222.patch
          47 kB
          jay vyas
        15. BIGTOP-1222.patch
          43 kB
          jay vyas
        16. BIGTOP-1222.patch
          51 kB
          jay vyas
        17. BIGTOP-1222.patch
          51 kB
          jay vyas
        18. BIGTOP-1222.patch
          50 kB
          jay vyas
        19. BIGTOP-1222.patch
          49 kB
          jay vyas
        20. BIGTOP-1222.patch
          49 kB
          jay vyas

        Issue Links

          Activity

            People

              cos Konstantin I Boudnik
              jayunit100 jay vyas
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: