Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Won't Do
    • None
    • None
    • CI

    Description

      Github Actions allows grouping output of workflow steps. We already use it in the build-info job (selective checks script). The goal of this change is to group output in the other CI checks to improve usability.

      Attachments

        Activity

          Helper functions to start/end groups are defined in _start_end.sh. To use these helpers, we need to source the script, along with some prerequisites, like this:

          local root_dir="${_testlib_dir}/../../../../.."
          source "${root_dir}/dev-support/ci/lib/_all_libs.sh"
          

          (Definition of root_dir will be different in various scripts, depending on where they are in the source tree.)

          After sourcing the libs, group can be created like this:

          start_end::group_start "<name of group, shown in collapsible item>"
          ...
          <code enclosed in group>
          ...
          start_end::group_end
          
          adoroszlai Attila Doroszlai added a comment - Helper functions to start/end groups are defined in _start_end.sh . To use these helpers, we need to source the script, along with some prerequisites, like this: local root_dir= "${_testlib_dir}/../../../../.." source "${root_dir}/dev-support/ci/lib/_all_libs.sh" (Definition of root_dir will be different in various scripts, depending on where they are in the source tree.) After sourcing the libs, group can be created like this: start_end::group_start "<name of group, shown in collapsible item>" ... <code enclosed in group> ... start_end::group_end

          Suggestions for grouping items:

          • acceptance
            1. installation of dependencies (code)
            2. execution of each test script (code)
            3. log collection (code)
          • integration/unit
            1. optional build (code)
            2. test (code)

          We can come up with more when these are done.

          adoroszlai Attila Doroszlai added a comment - Suggestions for grouping items: acceptance installation of dependencies ( code ) execution of each test script ( code ) log collection ( code ) integration/unit optional build ( code ) test ( code ) We can come up with more when these are done.
          himanshi.darvekar Himanshi Darvekar added a comment - - edited

          Thank you adoroszlai for pointing out the above scripts.

          For all the above scripts. I did the modification. All are working fine except for execution of each test script in testlib.sh  few CI checks are getting failed. I am trying to find reason here is my commit CI runs: https://github.com/HD372000/ozone/actions/runs/6230999266/job/16912253449#step:5:177  Can you also check it once?

          himanshi.darvekar Himanshi Darvekar added a comment - - edited Thank you adoroszlai for pointing out the above scripts. For all the above scripts. I did the modification. All are working fine except for execution of each test script in testlib.sh  few CI checks are getting failed. I am trying to find reason here is my commit CI runs: https://github.com/HD372000/ozone/actions/runs/6230999266/job/16912253449#step:5:177   Can you also check it once?
          Executing test upgrade/test.sh
            /home/runner/work/ozone/ozone/hadoop-ozone/dist/target/ozone-1.4.0-SNAPSHOT/compose/../../../../../dev-support/ci/lib/_all_libs.sh: line 19: LIBRARIES_DIR: readonly variable
            ERROR: Test execution of upgrade/test.sh is FAILED!!!!
          

          I think the problem is that _all_libs.sh does not like being sourced multiple times. testlib.sh sources it, and some scripts source testlib.sh multiple times (for re-initializing with different variables).

          This can be solved by checking whether some definition coming from _all_libs.sh exists already, something like:

          if ! type -f start_end::group_start >& /dev/null; then
            source "${root_dir}/dev-support/ci/lib/_all_libs.sh"
          fi
          
          adoroszlai Attila Doroszlai added a comment - Executing test upgrade/test.sh /home/runner/work/ozone/ozone/hadoop-ozone/dist/target/ozone-1.4.0-SNAPSHOT/compose/../../../../../dev-support/ci/lib/_all_libs.sh: line 19: LIBRARIES_DIR: readonly variable ERROR: Test execution of upgrade/test.sh is FAILED!!!! I think the problem is that _all_libs.sh does not like being sourced multiple times. testlib.sh sources it, and some scripts source testlib.sh multiple times (for re-initializing with different variables). This can be solved by checking whether some definition coming from _all_libs.sh exists already, something like: if ! type -f start_end::group_start >& /dev/ null ; then source "${root_dir}/dev-support/ci/lib/_all_libs.sh" fi

          If we source _all_libs.sh from inside the run_test_script() function then also the issue get solved.

          run_test_script() {
            local root_dir="${_testlib_dir}/../../../../.."
            source "${root_dir}/dev-support/ci/lib/_all_libs.sh"
            ...
          }

          So now all checks are getting passed https://github.com/HD372000/ozone/actions/runs/6244357976/job/16954081383 
          and the outputs are grouped.

          we can now move to next set of scripts. 

          himanshi.darvekar Himanshi Darvekar added a comment - If we source _all_libs.sh from inside the run_test_script() function then also the issue get solved. run_test_script() { local root_dir= "${_testlib_dir}/../../../../.." source "${root_dir}/dev-support/ci/lib/_all_libs.sh" ... } So now all checks are getting passed https://github.com/HD372000/ozone/actions/runs/6244357976/job/16954081383   and the outputs are grouped. we can now move to next set of scripts. 

          Thanks himanshi.darvekar for your work on this.

          Unfortunately, it seems scrolling doesn't work properly in the Github UI when long blocks of output are grouped. Expanding a collapsed group and trying to scroll makes it jump back and forth. So we may need to find another way of reducing the clutter of CI output.

          adoroszlai Attila Doroszlai added a comment - Thanks himanshi.darvekar for your work on this. Unfortunately, it seems scrolling doesn't work properly in the Github UI when long blocks of output are grouped. Expanding a collapsed group and trying to scroll makes it jump back and forth. So we may need to find another way of reducing the clutter of CI output.

          People

            himanshi.darvekar Himanshi Darvekar
            adoroszlai Attila Doroszlai
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: