Uploaded image for project: 'Daffodil'
  1. Daffodil
  2. DAFFODIL-2811

sbt IntegrationTest being deprecated, need alternative for slow CLI tests

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 3.5.0
    • CLI, Infrastructure
    • None

    Description

      SBT 1.9.0 is planning on deprecating the IntegrationTest configuration:

      https://eed3si9n.com/sbt-1.9.0-beta

      We currently use this config for slow running CLI tests that fork. With the upcoming release of 1.9.0, we may want to consider an alternative. Some requirements for the alternative:

      • Should allow running the tests in an IDE, which is currently difficult/impossible since not all IDE's support the IntegrationTest config
      • The CLI integration tests depend on the "daffodil-cli/stage" task since they actually execute the binary. Whatever alternative we choose should not run that task unless integration tests are actually run since staging the CLI can take awhile. We don't want to slow down the normal test cycle
      • We need the ability to disable parallelization when running these tests, otherwise we can quickly run out of memory on systems with limited resources (e.g. GitHub actions)
      • The tests should not run by default when running "sbt test". They are expensive and time consuming and usually don't need to be run as frequently as normal tests.

      Attachments

        Activity

          dfthompson Dave Thompson added a comment -

          Verified the specified commit (commit d524759ea7a7ec8e5eacee1614d3877df3d756ea) is included in the latest pull from the daffodil repository.

          Verified, via review, changes identified in the commit comment were implemented. 

          Verified file TestCLIUdfs.scala is no longer under daffodil-cli subproject and move to the new daffodil-test-integration subproject.
          Verified the command sbt IntegrationTest/test no longer executes the udf test suite. 
          Verified the command sbt daffodil-cli/it:test no longer executes the udf test suite. 
          Verified the command sbt daffodil-test-integration/test now executes the udf test suite.

          dfthompson Dave Thompson added a comment - Verified the specified commit (commit d524759ea7a7ec8e5eacee1614d3877df3d756ea) is included in the latest pull from the daffodil repository. Verified, via review, changes identified in the commit comment were implemented.  Verified file TestCLIUdfs.scala is no longer under daffodil-cli subproject and move to the new daffodil-test-integration subproject. Verified the command sbt IntegrationTest/test no longer executes the udf test suite.  Verified the command sbt daffodil-cli/it:test no longer executes the udf test suite.  Verified the command sbt daffodil-test-integration/test now executes the udf test suite.
          slawrence Steve Lawrence added a comment -

          Fixed in commit d524759ea7a7ec8e5eacee1614d3877df3d756ea

          slawrence Steve Lawrence added a comment - Fixed in commit d524759ea7a7ec8e5eacee1614d3877df3d756ea
          slawrence Steve Lawrence added a comment -

          RC2 has been released, and the release notes include a suggestion for migration away:

          https://eed3si9n.com/sbt-1.9.0-beta#deprecation-of-integrationtest-configuration

          Based on that, which I think meets all the requirements listed above, we could create a "daffodil-test-integration" subproject which is not published or aggregated, and move all slow forking CLI tests into that project. Any other slow tests can be move there as well if we detect them. IDE's can be configured to ignore this subproject if they do not want to run these tests, but can run them just like any other test. Our CI would be modified to run "sbt daffodil-test-integration/test" instead of "sbt IntegrationTest/test". And we can configure the subproject to disable parallelized tests.

          slawrence Steve Lawrence added a comment - RC2 has been released, and the release notes include a suggestion for migration away: https://eed3si9n.com/sbt-1.9.0-beta#deprecation-of-integrationtest-configuration Based on that, which I think meets all the requirements listed above, we could create a "daffodil-test-integration" subproject which is not published or aggregated, and move all slow forking CLI tests into that project. Any other slow tests can be move there as well if we detect them. IDE's can be configured to ignore this subproject if they do not want to run these tests, but can run them just like any other test. Our CI would be modified to run "sbt daffodil-test-integration/test" instead of "sbt IntegrationTest/test". And we can configure the subproject to disable parallelized tests.

          People

            Unassigned Unassigned
            slawrence Steve Lawrence
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: