• Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.9.0
    • 3.0.0-beta1
    • fs/azure
    • None


      The Azure tests are slow to run as they are serialized, as they are all called Test* there's no clear differentiation from unit tests which Jenkins can run, and integration tests which it can't.

      Move the azure tests Test* to integration tests ITest*, parallelize (which includes having separate paths for every test suite). The code in hadoop-aws's POM &c show what to do.

      UPDATE August 4, 2017: Adding a list of requirements to clarify the acceptance criteria for this JIRA:

      1. Parallelize test execution
      2. Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) IntegrationTests - requires Azure Storage account, executed by engineers prior to check-in, and if needed, iii) ScaleTests – long running performance and scalability tests.
      3. Define configuration profiles to run tests with different settings. Allows an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and false. Need to review settings to see what else would benefit.
      4. Maven commands to run b) and c). Turns out it is not easy to do with Maven, so we might have to run it multiple times to run with different configuration settings.
      5. Document how to add and run tests and the process for contributing to Apache Hadoop. Steve shared an example at https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
      6. UnitTests should run in under 2 minutes and IntegrationTests should run in under 15 minutes, even on slower network connections. (These are rough goals)
      7. Ensure test data (containers/blobs/etc) is deleted. Exceptions for large persistent content used repeatedly to expedite test execution.


        1. HADOOP-14553-001.patch
          92 kB
          Steve Loughran
        2. HADOOP-14553-002.patch
          120 kB
          Steve Loughran
        3. HADOOP-14553-003.patch
          120 kB
          Steve Loughran
        4. HADOOP-14553-004.patch
          178 kB
          Steve Loughran
        5. HADOOP-14553-005.patch
          239 kB
          Steve Loughran
        6. HADOOP-14553-006.patch
          259 kB
          Steve Loughran
        7. HADOOP-14553-007.patch
          260 kB
          Steve Loughran
        8. HADOOP-14553-008.patch
          264 kB
          Steve Loughran
        9. HADOOP-14553-009.patch
          288 kB
          Steve Loughran
        10. HADOOP-14553-010.patch
          293 kB
          Steve Loughran
        11. HADOOP-14553-011.patch
          294 kB
          Steve Loughran
        12. HADOOP-14553-012.patch
          296 kB
          Steve Loughran
        13. HADOOP-14553-014.patch
          299 kB
          Steve Loughran
        14. HADOOP-14553-015.patch
          301 kB
          Steve Loughran
        15. HADOOP-14553-016.patch
          300 kB
          Steve Loughran

        Issue Links



              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              0 Vote for this issue
              7 Start watching this issue