Details

    • Type: Sub-task Sub-task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.7.0
    • Fix Version/s: backlog
    • Component/s: build, tests
    • Labels:
      None

      Description

      This was discussed at the bigtop meetup 6/9/2014 at Redhat. This is related to JIrs: https://issues.apache.org/jira/browse/BIGTOP-1222

      Idea is to take currently what are hard coded parameters and test cases in the bigtop/bigtop-smoke-tests/build.gradle file and have it read from a parameter file a run/build time.

      Parameter file could be either Yaml, JSON, etc. Yaml is most human readable, might not be issue though since seems like there shouldnt be too much data residing in it

        Issue Links

          Activity

          Nate DAmico created issue -
          Nate DAmico made changes -
          Field Original Value New Value
          Parent BIGTOP-1222 [ 12697226 ]
          Issue Type Improvement [ 4 ] Sub-task [ 7 ]
          jay vyas made changes -
          Link This issue is blocked by BIGTOP-1222 [ BIGTOP-1222 ]
          Hide
          jay vyas added a comment -

          Linking to BIGTOP-1222

          Show
          jay vyas added a comment - Linking to BIGTOP-1222
          Hide
          jay vyas added a comment -

          Thanks - this is a great idea. so there will be a json file that is controller for all tests in BIGTOP-1222 - and gradle will use parameters from it.

          Show
          jay vyas added a comment - Thanks - this is a great idea. so there will be a json file that is controller for all tests in BIGTOP-1222 - and gradle will use parameters from it.
          Hide
          Nate DAmico added a comment -

          Should probably start by deciding on the params/values we want to have dynamically read at runtime. Currently see these as initial candidates in existing file. Possibly some new ones to be added

          ---------------------------------

          List of dependencies(Question: where there be variable dependencies based on what tests are run? Or assumption include kitchen sink?):

          dependencies

          { //needed to avoid groovy not on classpath error. testCompile module('org.codehaus.groovy:groovy:1.8.0') testCompile group:'org.apache.bigtop.itest', name:'itest-common', version:'0.7.0',transitive:'true' testCompile group:'org.apache.hadoop',name:'hadoop-common', version:'2.0.6-alpha',transitive:'true' testCompile group:'org.apache.hadoop',name:'hadoop-hdfs', version:'2.1.0-beta',transitive:'true' }

          -----------------------------------------------

          Tests to be run:

          def tests_to_include()

          { return [ //"TestHadoopExamples.groovy", //"TestHiveSmokeBulk.groovy", //"TestHiveSmokeBulk.groovy", //"HiveBulkScriptExecutor.groovy", //"TestPigSmoke.groovy", 'TestMahoutExamples.groovy' ] }

          NOTE: seems the doExclude and sourceSets will need updating based on the tests that are included

          ---------------------------------------

          Environment Vars:

          def vars = [
          "PIG_HOME",
          "HIVE_HOME",
          "HIVE_CONF_DIR",
          "HADOOP_CONF_DIR",
          "BIGTOP_HOME", // for copying input files / resources
          "HADOOP_MAPRED_HOME" // required for mapreduce programs to run
          ];

          Seems these could probably extended to have some defaults, dynamically read others. Probably would want to extend to read from environment and if not present try to read from "external source", lastly if not found throw exception as do now

          Show
          Nate DAmico added a comment - Should probably start by deciding on the params/values we want to have dynamically read at runtime. Currently see these as initial candidates in existing file. Possibly some new ones to be added --------------------------------- List of dependencies(Question: where there be variable dependencies based on what tests are run? Or assumption include kitchen sink?): dependencies { //needed to avoid groovy not on classpath error. testCompile module('org.codehaus.groovy:groovy:1.8.0') testCompile group:'org.apache.bigtop.itest', name:'itest-common', version:'0.7.0',transitive:'true' testCompile group:'org.apache.hadoop',name:'hadoop-common', version:'2.0.6-alpha',transitive:'true' testCompile group:'org.apache.hadoop',name:'hadoop-hdfs', version:'2.1.0-beta',transitive:'true' } ----------------------------------------------- Tests to be run: def tests_to_include() { return [ //"TestHadoopExamples.groovy", //"TestHiveSmokeBulk.groovy", //"TestHiveSmokeBulk.groovy", //"HiveBulkScriptExecutor.groovy", //"TestPigSmoke.groovy", 'TestMahoutExamples.groovy' ] } NOTE: seems the doExclude and sourceSets will need updating based on the tests that are included --------------------------------------- Environment Vars: def vars = [ "PIG_HOME", "HIVE_HOME", "HIVE_CONF_DIR", "HADOOP_CONF_DIR", "BIGTOP_HOME", // for copying input files / resources "HADOOP_MAPRED_HOME" // required for mapreduce programs to run ]; Seems these could probably extended to have some defaults, dynamically read others. Probably would want to extend to read from environment and if not present try to read from "external source", lastly if not found throw exception as do now
          Hide
          Konstantin Boudnik added a comment -

          Off the top of my head:
          HBASE_HOME
          PHOENIX_HOME
          SQOOP_HOME
          SPARK_HOME
          ZOOKEEPER_HOME

          Crunch also expects
          HADOOP_MAPRED_HOME

          Now sure why do we need BIGTOP_HOME though?

          Show
          Konstantin Boudnik added a comment - Off the top of my head: HBASE_HOME PHOENIX_HOME SQOOP_HOME SPARK_HOME ZOOKEEPER_HOME Crunch also expects HADOOP_MAPRED_HOME Now sure why do we need BIGTOP_HOME though?
          Hide
          Nate DAmico added a comment -

          not sure about BIGTOP_HOME, maybe jay vyas added for test data and future use of the bigpetstore effort

          there is this comment in current build.gradle file: // for copying input files / resources

          Show
          Nate DAmico added a comment - not sure about BIGTOP_HOME, maybe jay vyas added for test data and future use of the bigpetstore effort there is this comment in current build.gradle file: // for copying input files / resources
          Hide
          jay vyas added a comment -

          Hi nate: iirc I store BIGTOP_HOME Because in BIGTOP-1222 I'm pulling in the source directly for some tests - so need to know where the root directory is.

          Show
          jay vyas added a comment - Hi nate: iirc I store BIGTOP_HOME Because in BIGTOP-1222 I'm pulling in the source directly for some tests - so need to know where the root directory is.

            People

            • Assignee:
              Unassigned
              Reporter:
              Nate DAmico
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development