Uploaded image for project: 'Hadoop Distributed Data Store'
  1. Hadoop Distributed Data Store
  2. HDDS-1424

Support multi-container robot test execution

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.1
    • Component/s: test
    • Target Version/s:
    • Sprint:
      HDDS Biscayne

      Description

      The ./smoketest folder in the distribution package contains robotframework based test scripts to test the main behaviour of Ozone.

      The tests have two layers:

      1. robot test definitions to execute commands and assert the results (on a given host machine)
      2. ./smoketest/test.sh which starts/stops the docker-compose based environments AND execute the selected robot tests inside the right hosts

      The second one (test.sh) has some serious limitations:

      1. all the tests are executed inside the same container (om):

      https://github.com/apache/hadoop/blob/5f951ea2e39ae4dfe554942baeec05849cd7d3c2/hadoop-ozone/dist/src/main/smoketest/test.sh#L89

      Some of the tests (ozonesecure-mr, ozonefs) may require the flexibility to execute different robot tests in different containers.

      2. The definition of the global test set is complex and hard to understood.

      The current code is:

         TESTS=("basic")
         execute_tests ozone "${TESTS[@]}"
         TESTS=("auditparser")
         execute_tests ozone "${TESTS[@]}"
         TESTS=("ozonefs")
         execute_tests ozonefs "${TESTS[@]}"
         TESTS=("basic")
         execute_tests ozone-hdfs "${TESTS[@]}"
         TESTS=("s3")
         execute_tests ozones3 "${TESTS[@]}"
         TESTS=("security")
         execute_tests ozonesecure .
      

      For example for ozonesecure the TESTS is not used. And the usage of bash lists require additional complexity in the execute_tests function.

      I propose here a very lightweight refactor. Instead of including both the test definitions AND the helper methods in test.sh I would separate them.

      Let's put a test.sh to each of the compose directories. The separated test.sh can include common methods from a main shell script. For example:

      
      source "$COMPOSE_DIR/../testlib.sh"
      
      start_docker_env
      
      execute_robot_test scm basic/basic.robot
      
      execute_robot_test scm s3
      
      stop_docker_env
      
      generate_report
      
      

      This is a more clean and more flexible definition. It's easy to execute just this test (as it's saved to the compose/ozones3 directory. And it's more flexible.

      Other example, where multiple containers are used to execute tests:

      
      source "$COMPOSE_DIR/../testlib.sh"
      
      start_docker_env
      
      execute_robot_test scm ozonefs/ozonefs.robot
      
      
      
      export OZONE_HOME=/opt/ozone
      
      execute_robot_test hadoop32 ozonefs/hadoopo3fs.robot
      
      execute_robot_test hadoop31 ozonefs/hadoopo3fs.robot
      
      stop_docker_env
      
      generate_report
      

      With this separation the definition of the helper methods (eg. execute_robot_test or stop_docker_env) would also be simplified.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                elek Marton Elek
                Reporter:
                elek Marton Elek
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3.5h
                  3.5h