Bigtop
  1. Bigtop
  2. BIGTOP-476

Improvement of BigTop iTest framework

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.3.0
    • Fix Version/s: None
    • Component/s: Tests
    • Labels:
    • Environment:

      All

      Description

      Current BigTop test framework has a limitation to handle dynamic generated data. It's flexibility can be improved.

      For org.apache.bigtop.itest.hadoopexamples.TestHadoopExamples

      Limitation: if someone wants to make any changes, he/she needs to modify
      ./bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/TestHadoopExamples.groovy. It requires compilation before running.

      For org.apache.bigtop.itest.hadooptests.TestTestCLI. The configuration file,
      ./build/hadoop/deb/hadoop-1.0.1/src/test/org/apache/hadoop/cli/testConf.xml, has entries like the following:

      <test> <!-- TESTED -->
      <description>ls: file using relative path</description>
      <test-commands>
      <command>-fs NAMENODE -touchz file1</command>
      <command>-fs NAMENODE -ls file1</command>
      </test-commands>
      <cleanup-commands>
      <command>-fs NAMENODE -rm file1</command>
      </cleanup-commands>
      <comparators>
      <comparator>
      <type>TokenComparator</type>
      <expected-output>Found 1 items</expected-output>
      </comparator>
      <comparator>
      <type>RegexpComparator</type>
      <expected-output>^rw-rr-( )1( )[a-z]( )*supergroup( )*0( )[0-9]

      {4,}

      -[0-9]

      {2,}-[0-9]{2,}

      [0-9]

      {2,}:[0-9]{2,}

      ( )/user/[a-z]/file1</expected-output>
      </comparator>
      </comparators>
      </test>

      Limitation: Put the expected-output, then perform string comparison is good, but still not flexible enough to handle dynamic generated data. For example, a program randomly generate key/value pairs, then submit M/R job to calculate sum (average) for each key. There is no way to calculate the result in advance to put down as expected-output.

      I am proposing an improvement for BigTop's integration test. We can put all test cases in a XML file, which contains a list of command-sets; each command-set has command, command-comparator-type, command-comparator-compare-to. The command is for hadoop/hbase/hive command; command-comparator-type to specify Java class to perform comparison; command-comparator-compare-to is used to specify the shell command to generate expected output.

      I put down 3 cases below:

      <?xml version="1.0" encoding="ISO-8859-1"?>
      <bigtop-itest-suite>
      <bigtop-itest-suite-test>
      <test-name>Calculate summation in MR</test-name>
      <test-desc>Here is simple MR test to calculate sum</test-desc>
      <test-pre-integration-test>
      </test-pre-integration-test>
      <test-integration-test>
      <command-set>
      <command>hadoop jar ./target/LeiBigTop-1.1.jar com.lei.bigtop.hadoop.calsum.CalSum ./data ./output</command>
      <command-comparator-type>com.lei.bigtop.hadoop.integration.test.ExtactComparatorIgnoreWhiteSpace</command-comparator-type>
      <command-comparator-compare-to><![CDATA[ cat ./output/* ]]></command-comparator-compare-to>
      </command-set>
      </test-integration-test>
      <test-post-integration-test>
      </test-post-integration-test>
      </bigtop-itest-suite-test>

      <bigtop-itest-suite-test>
      <test-name>calculate pi</test-name>
      <test-desc>calculate pi using hadoop MR</test-desc>
      <test-pre-integration-test>
      </test-pre-integration-test>
      <test-integration-test>
      <command-set>
      <command>hadoop jar $HADOOP_HOME/hadoop-examples-0.*.0.jar pi 5 5</command>
      <command-comparator-type>org.apache.hadoop.cli.util.SubstringComparator</command-comparator-type>
      <command-comparator-compare-to><![CDATA[echo "Pi is 3.68"]]></command-comparator-compare-to>
      </command-set>
      </test-integration-test>
      <test-post-integration-test>
      </test-post-integration-test>
      </bigtop-itest-suite-test>

      <bigtop-itest-suite-test>
      <test-name>count word in MR</test-name>
      <test-desc>count word in Hadoop MR</test-desc>
      <test-pre-integration-test>
      <command-set><command>rm -rf ./wordcount</command></command-set>
      <command-set><command>rm -rf ./wordcount_out</command></command-set>
      <command-set><command>mkdir ./wordcount</command></command-set>
      <command-set><command><![CDATA[curl http://www.meetup.com/HandsOnProgrammingEvents/events/53837022/ | sed -e :a -e 's/<[^>]>//g;/</N;//ba' | sed 's/&nbsp//g' | sed 's/[ \t]//;s/[ \t]$//' | sed '/$/d' | sed '/"http[^"]"/d' > ./wordcount/content]]></command></command-set>
      <command-set><command>hadoop fs -mkdir /wordcount</command></command-set>
      <command-set><command>hadoop fs -put ./wordcount/* /wordcount</command></command-set>
      </test-pre-integration-test>
      <test-integration-test>
      <command-set><command>hadoop jar $HADOOP_HOME/hadoop-examples-0.*.0.jar wordcount /wordcount /wordcount_out</command></command-set>
      <command-set><command>mkdir ./wordcount_out</command></command-set>
      <command-set><command>hadoop fs -get /wordcount_out/* ./wordcount_out</command></command-set>
      <command-set><command>hadoop fs -rmr /wordcount</command></command-set>
      <command-set><command>hadoop fs -rmr /wordcount_out/</command></command-set>
      </test-integration-test>
      <test-post-integration-test>
      <command-set>
      <command>cat ./wordcount_out/* | grep Roman | sed 's/[^0-9.]([0-9.]).*/\1/'</command>
      <command-comparator-type>com.lei.bigtop.hadoop.integration.test.ExtactComparatorIgnoreWhiteSpace</command-comparator-type>
      <command-comparator-compare-to><![CDATA[cat wordcount/* | grep -c Roman]]></command-comparator-compare-to>
      </command-set>
      </test-post-integration-test>
      </bigtop-itest-suite-test>

      </bigtop-itest-suite>

      1. bigtop-tests_test-artifacts_pom.xml.diff
        0.8 kB
        Lei Zou
      2. bigtop-tests_test-execution_smokes_hadoop_pom.xml.diff
        2 kB
        Lei Zou
      3. BigTopIntegrationTest.java
        5 kB
        Lei Zou
      4. BigTopIntegrationTestFacade.groovy
        2 kB
        Lei Zou
      5. BigTopTestCommand.java
        2 kB
        Lei Zou
      6. ExtactComparatorIgnoreWhiteSpace.groovy
        1 kB
        Lei Zou
      7. TestRunHadoopExamples.groovy
        8 kB
        Lei Zou
      8. bigtop-testcases.yaml
        8 kB
        Lei Zou

        Activity

        Hide
        Konstantin Boudnik added a comment -

        Actually, I think writing tests in XML is equally lame as doing it in YAML. Perhaps XML is a bit better. What are you trying to solve here? Avoid the compilation step or something?

        Show
        Konstantin Boudnik added a comment - Actually, I think writing tests in XML is equally lame as doing it in YAML. Perhaps XML is a bit better. What are you trying to solve here? Avoid the compilation step or something?
        Hide
        Stephen Chu added a comment -

        Hi Lei, you might be interested in the framework Sujay Rau is working on in https://issues.apache.org/jira/browse/BIGTOP-635. It's related to your question.

        Show
        Stephen Chu added a comment - Hi Lei, you might be interested in the framework Sujay Rau is working on in https://issues.apache.org/jira/browse/BIGTOP-635 . It's related to your question.
        Hide
        Lei Zou added a comment -

        Yes, I am interested in this.

        Show
        Lei Zou added a comment - Yes, I am interested in this.
        Hide
        Roman Shaposhnik added a comment -

        Lei, what you're referring to would be an extremely desirable functionality to have. Currently there's no single API that would allow you to do that short of ssh-based script (ugly and unreliable). The last effort in that area was the Herriot framework: http://wiki.apache.org/hadoop/HowToUseSystemTestFramework

        If you're interested in reviving that effort, I'm sure you'll get quiet a bit of help.

        Show
        Roman Shaposhnik added a comment - Lei, what you're referring to would be an extremely desirable functionality to have. Currently there's no single API that would allow you to do that short of ssh-based script (ugly and unreliable). The last effort in that area was the Herriot framework: http://wiki.apache.org/hadoop/HowToUseSystemTestFramework If you're interested in reviving that effort, I'm sure you'll get quiet a bit of help.
        Hide
        Lei Zou added a comment - - edited

        Thanks for your time for review and guidance. Putting the code under org.apache.bigtop.itest.executors.cli makes sense to me, we can better leverage the same functionality. I will take a closer look at the BigTop's architecture/packaging and see how this piece would fit.

        I have another question in my mind, and I put it out here: in the land of Hadoop clustered environment after the successful BigTop's deployment, how does BigTop check and see if all multiple datanodes (desired) participate in M/R job in a programmable way? For example, let's say we have one namenode and two datadanodes in Hadoop cluster, we like to see all three servers participate in the Hadoop job that calculates Pi. Verification is a success if all nodes have done some work, fail if anyone was not involved.

        Show
        Lei Zou added a comment - - edited Thanks for your time for review and guidance. Putting the code under org.apache.bigtop.itest.executors.cli makes sense to me, we can better leverage the same functionality. I will take a closer look at the BigTop's architecture/packaging and see how this piece would fit. I have another question in my mind, and I put it out here: in the land of Hadoop clustered environment after the successful BigTop's deployment, how does BigTop check and see if all multiple datanodes (desired) participate in M/R job in a programmable way? For example, let's say we have one namenode and two datadanodes in Hadoop cluster, we like to see all three servers participate in the Hadoop job that calculates Pi. Verification is a success if all nodes have done some work, fail if anyone was not involved.
        Hide
        Roman Shaposhnik added a comment -

        Lei, sorry it took me so long to finally review your stuff. I finally did and at this point I really would like include this into the iTest framework itself (under bigtop-test-framework). I would imagine that we can create a brand new hierarchy of org.apache.bigtop.itest.executors and have your stuff under org.apache.bigtop.itest.executors.cli.

        Would you be interested in working on an overall patch that would integrate your lose java/grovy files in that way? Then we can simply leverage it in any of the test artifacts that have to exercise command line functionality.

        Show
        Roman Shaposhnik added a comment - Lei, sorry it took me so long to finally review your stuff. I finally did and at this point I really would like include this into the iTest framework itself (under bigtop-test-framework). I would imagine that we can create a brand new hierarchy of org.apache.bigtop.itest.executors and have your stuff under org.apache.bigtop.itest.executors.cli. Would you be interested in working on an overall patch that would integrate your lose java/grovy files in that way? Then we can simply leverage it in any of the test artifacts that have to exercise command line functionality.
        Hide
        Lei Zou added a comment -

        Thanks for feedbacks from Wing Yew and Roman, I was set out to improve

        ./bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/TestHadoopExamples.groovy

        using YAML, Groovy and existing itest infrastructure.

        Here are the two areas, I have made the improvement.

        1. Move the commands inside TestHadoopExamples.groovy to YAML file, so that we need NOT compile the code if add/change the test cases.

        2. Introduce the comparator to compare output with the expected to verify the commands' execution. For example, when calculating Pi, make sure the number matched to the known result.

        The code has been tested and submitted.

        Show
        Lei Zou added a comment - Thanks for feedbacks from Wing Yew and Roman, I was set out to improve ./bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/TestHadoopExamples.groovy using YAML, Groovy and existing itest infrastructure. Here are the two areas, I have made the improvement. 1. Move the commands inside TestHadoopExamples.groovy to YAML file, so that we need NOT compile the code if add/change the test cases. 2. Introduce the comparator to compare output with the expected to verify the commands' execution. For example, when calculating Pi, make sure the number matched to the known result. The code has been tested and submitted.
        Hide
        Lei Zou added a comment -

        Here are the list of new files:

        1. new file: bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/BigTopIntegrationTest.java
        2. new file: bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/BigTopIntegrationTestFacade.groovy
        3. new file: bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/BigTopTestCommand.java
        4. new file: bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/ExtactComparatorIgnoreWhiteSpace.groovy
        5. new file: bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/TestRunHadoopExamples.groovy
        6. new file: bigtop-tests/test-execution/smokes/hadoop/bigtop-testcases.yaml

        Here are the list of modified files:

        1. modified: bigtop-tests/test-artifacts/pom.xml
        2. modified: bigtop-tests/test-execution/smokes/hadoop/pom.xml
        Show
        Lei Zou added a comment - Here are the list of new files: new file: bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/BigTopIntegrationTest.java new file: bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/BigTopIntegrationTestFacade.groovy new file: bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/BigTopTestCommand.java new file: bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/ExtactComparatorIgnoreWhiteSpace.groovy new file: bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/TestRunHadoopExamples.groovy new file: bigtop-tests/test-execution/smokes/hadoop/bigtop-testcases.yaml Here are the list of modified files: modified: bigtop-tests/test-artifacts/pom.xml modified: bigtop-tests/test-execution/smokes/hadoop/pom.xml
        Hide
        Lei Zou added a comment -

        For improvement No. 1, Here is the sample YAML

        • !!org.apache.bigtop.itest.hadoopexamples.BigTopIntegrationTest
          integrationTestCommandList:
        • !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop jar ./target/LeiBigTop-1.1.jar com.lei.bigtop.hadoop.calsum.CalSum ./data ./output, commandComparator: ' cat ./output/* ', comparatorClass: org.apache.bigtop.itest.hadoopexamples.ExtactComparatorIgnoreWhiteSpace}

          postTestCommandList: []
          preTestCommandList: []
          testDesc: Here is simple MR test to calculate sum
          testName: Calculate summation in MR

        • !!org.apache.bigtop.itest.hadoopexamples.BigTopIntegrationTest
          integrationTestCommandList:
        • !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop jar $HADOOP_HOME/hadoop-examples-0.*.0.jar pi 5 5, commandComparator: echo "Pi is 3.68", comparatorClass: org.apache.hadoop.cli.util.SubstringComparator}

          postTestCommandList: []
          preTestCommandList: []
          testDesc: calculate pi using hadoop MR
          testName: calculate pi

        • !!org.apache.bigtop.itest.hadoopexamples.BigTopIntegrationTest
          integrationTestCommandList:
        • !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop jar $HADOOP_HOME/hadoop-examples-0.*.0.jar wordcount /wordcount /wordcount_out, commandComparator: null, comparatorClass: null}
        • !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: mkdir ./wordcount_out, commandComparator: null, comparatorClass: null}
        • !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -get /wordcount_out/* ./wordcount_out, commandComparator: null, comparatorClass: null}
        • !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -rmr /wordcount, commandComparator: null, comparatorClass: null}
        • !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -rmr /wordcount_out/, commandComparator: null, comparatorClass: null}

          postTestCommandList:

        • !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: 'cat ./wordcount_out/* | grep Roman | sed ''s/[^0-9.]*\([0-9.]*\).*/\1/''', commandComparator: cat wordcount/* | grep -c Roman, comparatorClass: org.apache.bigtop.itest.hadoopexamples.ExtactComparatorIgnoreWhiteSpace}

          preTestCommandList:

        • !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: rm -rf ./wordcount, commandComparator: null, comparatorClass: null}
        • !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: rm -rf ./wordcount_out, commandComparator: null, comparatorClass: null}
        • !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: mkdir ./wordcount, commandComparator: null, comparatorClass: null}
        • !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: 'curl http://www.meetup.com/HandsOnProgrammingEvents/events/53837022/ | sed -e :a -e ''s/<[^>]*>//g;/</N;//ba'' | sed ''s/&nbsp//g'' | sed ''s/^[ \t]*//;s/[ \t]*$//'' | sed ''/^$/d'' | sed ''/"http[^"]*"/d'' > ./wordcount/content', commandComparator: null, comparatorClass: null}
        • !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -mkdir /wordcount, commandComparator: null, comparatorClass: null}
        • !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -put ./wordcount/* /wordcount, commandComparator: null, comparatorClass: null}

          testDesc: count word in Hadoop MR
          testName: count word in MR

        Show
        Lei Zou added a comment - For improvement No. 1, Here is the sample YAML !!org.apache.bigtop.itest.hadoopexamples.BigTopIntegrationTest integrationTestCommandList: !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop jar ./target/LeiBigTop-1.1.jar com.lei.bigtop.hadoop.calsum.CalSum ./data ./output, commandComparator: ' cat ./output/* ', comparatorClass: org.apache.bigtop.itest.hadoopexamples.ExtactComparatorIgnoreWhiteSpace} postTestCommandList: [] preTestCommandList: [] testDesc: Here is simple MR test to calculate sum testName: Calculate summation in MR !!org.apache.bigtop.itest.hadoopexamples.BigTopIntegrationTest integrationTestCommandList: !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop jar $HADOOP_HOME/hadoop-examples-0.*.0.jar pi 5 5, commandComparator: echo "Pi is 3.68", comparatorClass: org.apache.hadoop.cli.util.SubstringComparator} postTestCommandList: [] preTestCommandList: [] testDesc: calculate pi using hadoop MR testName: calculate pi !!org.apache.bigtop.itest.hadoopexamples.BigTopIntegrationTest integrationTestCommandList: !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop jar $HADOOP_HOME/hadoop-examples-0.*.0.jar wordcount /wordcount /wordcount_out, commandComparator: null, comparatorClass: null} !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: mkdir ./wordcount_out, commandComparator: null, comparatorClass: null} !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -get /wordcount_out/* ./wordcount_out, commandComparator: null, comparatorClass: null} !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -rmr /wordcount, commandComparator: null, comparatorClass: null} !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -rmr /wordcount_out/, commandComparator: null, comparatorClass: null} postTestCommandList: !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: 'cat ./wordcount_out/* | grep Roman | sed ''s/[^0-9.]*\([0-9.]*\).*/\1/''', commandComparator: cat wordcount/* | grep -c Roman, comparatorClass: org.apache.bigtop.itest.hadoopexamples.ExtactComparatorIgnoreWhiteSpace} preTestCommandList: !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: rm -rf ./wordcount, commandComparator: null, comparatorClass: null} !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: rm -rf ./wordcount_out, commandComparator: null, comparatorClass: null} !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: mkdir ./wordcount, commandComparator: null, comparatorClass: null} !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: 'curl http://www.meetup.com/HandsOnProgrammingEvents/events/53837022/ | sed -e :a -e ''s/<[^>]*>//g;/</N;//ba'' | sed ''s/&nbsp//g'' | sed ''s/^[ \t]*//;s/[ \t]*$//'' | sed ''/^$/d'' | sed ''/"http[^"]*"/d'' > ./wordcount/content', commandComparator: null, comparatorClass: null} !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -mkdir /wordcount, commandComparator: null, comparatorClass: null} !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -put ./wordcount/* /wordcount, commandComparator: null, comparatorClass: null} testDesc: count word in Hadoop MR testName: count word in MR
        Hide
        Lei Zou added a comment -

        I agree with Wing Yew and Roman, YAML is better than XML in this situation. There should be 2 different improvements.

        1. Improve the current ./bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/TestHadoopExamples.groovy. We could replace it with YAML driven groovy class (RunTestHadoopExamples.groovy). It reads the commands from bigtop-testcases.yaml and run one command at the time, and compare the result with expected output.

        2. Improve the test granularity and failure isolation.

        Show
        Lei Zou added a comment - I agree with Wing Yew and Roman, YAML is better than XML in this situation. There should be 2 different improvements. 1. Improve the current ./bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/TestHadoopExamples.groovy. We could replace it with YAML driven groovy class (RunTestHadoopExamples.groovy). It reads the commands from bigtop-testcases.yaml and run one command at the time, and compare the result with expected output. 2. Improve the test granularity and failure isolation.
        Hide
        Roman Shaposhnik added a comment -

        I second Wing Yew's take on XML vs. more human-readable format. With Groovy we've got parsers/slurpers for all sorts of formats and they are all extremely easy to use so I don't think XML has an edge there.

        The second point is even more important. Test granularity and failure isolation are extremely useful. Whatever we end up doing it would really nice to have Jenkins report individual test cases failure in such a way as to make it easy to parse results in useful chunks.

        I recommend taking a look at bigtop-tests/test-artifacts/package/src/main/groovy/org/apache/bigtop/itest/packagesmoke/TestPackagesBasics.groovy (generateTests) for an example of how to generate multiple JUnit testcases from a single static manifest on the fly.

        Show
        Roman Shaposhnik added a comment - I second Wing Yew's take on XML vs. more human-readable format. With Groovy we've got parsers/slurpers for all sorts of formats and they are all extremely easy to use so I don't think XML has an edge there. The second point is even more important. Test granularity and failure isolation are extremely useful. Whatever we end up doing it would really nice to have Jenkins report individual test cases failure in such a way as to make it easy to parse results in useful chunks. I recommend taking a look at bigtop-tests/test-artifacts/package/src/main/groovy/org/apache/bigtop/itest/packagesmoke/TestPackagesBasics.groovy (generateTests) for an example of how to generate multiple JUnit testcases from a single static manifest on the fly.
        Hide
        Wing Yew Poon added a comment -

        I am not a fan of writing tests in XML.
        Using an external data file (which happens to be XML) in TestCLI makes some sense, in that it is about running many FsShell commands and parsing the results of the commands. It is specialized for running FsShell commands.
        Of course, the idea may be generalized to running arbitrary shell commands, but if I understood this proposal correctly, it is about building a framework for writing tests in XML by running arbitrary shell commands and then comparing the output.
        Even if such an idea has merit, I'd prefer that it uses YAML or a DSL, instead of XML.

        On the subject of TestCLI, it has a shortcoming, and that is it counts as s single JUnit test. In actual fact, it consists of more than a hundred testcases. If one test case or 100 test cases fail, it is reported as a single failure, which is not useful. This is something that could use improvement.

        Show
        Wing Yew Poon added a comment - I am not a fan of writing tests in XML. Using an external data file (which happens to be XML) in TestCLI makes some sense, in that it is about running many FsShell commands and parsing the results of the commands. It is specialized for running FsShell commands. Of course, the idea may be generalized to running arbitrary shell commands, but if I understood this proposal correctly, it is about building a framework for writing tests in XML by running arbitrary shell commands and then comparing the output. Even if such an idea has merit, I'd prefer that it uses YAML or a DSL, instead of XML. On the subject of TestCLI, it has a shortcoming, and that is it counts as s single JUnit test. In actual fact, it consists of more than a hundred testcases. If one test case or 100 test cases fail, it is reported as a single failure, which is not useful. This is something that could use improvement.

          People

          • Assignee:
            Lei Zou
            Reporter:
            Lei Zou
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Due:
              Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 168h
              168h
              Remaining:
              Remaining Estimate - 168h
              168h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development