Uploaded image for project: 'Apache Knox'
  1. Apache Knox
  2. KNOX-1004

Failing (flaky) Knox unit test: org.apache.hadoop.gateway.GatewayAppFuncTest.testDefaultTopology

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.14.0
    • Component/s: Tests
    • Labels:
      None
    • Environment:
      local machine

      Description

      Found flaky unit test in Knox:
      org.apache.hadoop.gateway.GatewayAppFuncTest.testDefaultTopology

      The problem occurs when the run is a little bit slower. The directory deletion during test cleanup fails due to missing directories. The directories are being deleted by Knox TopologyService.

      The problem in code where the second deletion is failing sometimes:

        @After
        public void cleanupTest() throws Exception {
          FileUtils.cleanDirectory( new File( config.getGatewayTopologyDir() ) );
          FileUtils.cleanDirectory( new File( config.getGatewayDeploymentDir() ) );
        }
      

      Due to removing working directories are being deleted during cleanupSuite I suggest remove deployment directory ignoring file-not-found errors. For example:

        @After
        public void cleanupTest() throws Exception {
          ...
          FileUtils.deleteQuietly( new File( config.getGatewayDeploymentDir() ) );
        }
      

      Reproduction:

      Place a breakpoint at the deployment deletion and then slowly follow the steps. During in a file browser the deletion can be seen.

        Activity

        Hide
        lmccay Larry McCay added a comment -

        Denes Bodo - do you plan to contribute a patch for this?

        Show
        lmccay Larry McCay added a comment - Denes Bodo - do you plan to contribute a patch for this?
        Hide
        dionusos Denes Bodo added a comment - - edited

        Larry McCay, yes, I would like to work on an acceptable fix.

        Show
        dionusos Denes Bodo added a comment - - edited Larry McCay , yes, I would like to work on an acceptable fix.
        Hide
        coheigea Colm O hEigeartaigh added a comment -

        Where exactly are the directories being deleted? You mentioned "cleanupSuite", but this is run with "@AfterClass" and so surely it should never run before the @After test method?

        BTW a similar pattern is in GatewaySslFuncTest and AmbariServiceDefinitionTest tests, so if we are fixing this then those tests should be included in the patch.

        Show
        coheigea Colm O hEigeartaigh added a comment - Where exactly are the directories being deleted? You mentioned "cleanupSuite", but this is run with "@AfterClass" and so surely it should never run before the @After test method? BTW a similar pattern is in GatewaySslFuncTest and AmbariServiceDefinitionTest tests, so if we are fixing this then those tests should be included in the patch.
        Hide
        dionusos Denes Bodo added a comment -

        Colm O hEigeartaigh Thank you for your question. The directories I am talking about are on path ./gateway-test/target/gateway-home{$ID}/deployments/* . I understand @After and @AfterClass precedency. What I would like to explain (I think) is that during test cleanUp (@After) a Knox service is also deleting the same directories which causes an Exception in cleanUp method.

        Also thank you for notifying me about similar patterns. I am going to fix them if the recommendation is acceptable.

        Show
        dionusos Denes Bodo added a comment - Colm O hEigeartaigh Thank you for your question. The directories I am talking about are on path ./gateway-test/target/gateway-home{$ID}/deployments/* . I understand @After and @AfterClass precedency. What I would like to explain (I think) is that during test cleanUp (@After) a Knox service is also deleting the same directories which causes an Exception in cleanUp method. Also thank you for notifying me about similar patterns. I am going to fix them if the recommendation is acceptable.
        Hide
        coheigea Colm O hEigeartaigh added a comment -

        Yes that's fine thanks, please fix the other tests as well and resubmit the patch.

        Show
        coheigea Colm O hEigeartaigh added a comment - Yes that's fine thanks, please fix the other tests as well and resubmit the patch.
        Hide
        dionusos Denes Bodo added a comment -

        Extended patch. Sorry for the delay.

        Show
        dionusos Denes Bodo added a comment - Extended patch. Sorry for the delay.
        Hide
        coheigea Colm O hEigeartaigh added a comment -

        The patch does not apply with the latest master - can you update your local repo and resubmit the patch please?

        Show
        coheigea Colm O hEigeartaigh added a comment - The patch does not apply with the latest master - can you update your local repo and resubmit the patch please?
        Hide
        dionusos Denes Bodo added a comment -

        Colm O hEigeartaigh I updated my local repo, checked out master, rebased and applied this patch without any conflict. Build and related tests are successful.

        Show
        dionusos Denes Bodo added a comment - Colm O hEigeartaigh I updated my local repo, checked out master, rebased and applied this patch without any conflict. Build and related tests are successful.
        Hide
        coheigea Colm O hEigeartaigh added a comment -

        How are you applying the patch? With "git apply" I get:

        git apply 0001-KNOX-1004-Failing-flaky-Knox-unit-test-org.apache.ha.patch
        error: patch failed: gateway-test/src/test/java/org/apache/hadoop/gateway/GatewayAppFuncTest.java:103
        error: gateway-test/src/test/java/org/apache/hadoop/gateway/GatewayAppFuncTest.java: patch does not apply
        error: patch failed: gateway-test/src/test/java/org/apache/hadoop/gateway/GatewaySslFuncTest.java:124
        error: gateway-test/src/test/java/org/apache/hadoop/gateway/GatewaySslFuncTest.java: patch does not apply

        Show
        coheigea Colm O hEigeartaigh added a comment - How are you applying the patch? With "git apply" I get: git apply 0001- KNOX-1004 -Failing-flaky-Knox-unit-test-org.apache.ha.patch error: patch failed: gateway-test/src/test/java/org/apache/hadoop/gateway/GatewayAppFuncTest.java:103 error: gateway-test/src/test/java/org/apache/hadoop/gateway/GatewayAppFuncTest.java: patch does not apply error: patch failed: gateway-test/src/test/java/org/apache/hadoop/gateway/GatewaySslFuncTest.java:124 error: gateway-test/src/test/java/org/apache/hadoop/gateway/GatewaySslFuncTest.java: patch does not apply
        Hide
        dionusos Denes Bodo added a comment -

        I used the 'git apply

        {path/to/patch}

        ' command exactly.

        Show
        dionusos Denes Bodo added a comment - I used the 'git apply {path/to/patch} ' command exactly.
        Hide
        coheigea Colm O hEigeartaigh added a comment -

        If you do "git log" on your repo, what is the last commit that you see?

        Show
        coheigea Colm O hEigeartaigh added a comment - If you do "git log" on your repo, what is the last commit that you see?
        Hide
        dionusos Denes Bodo added a comment -

        $ git log -1
        commit 773ac9d8e2594241d5d95448b19a75982f467efd
        Author: Colm O hEigeartaigh <coheigea@apache.org>
        Date: Mon Sep 11 10:29:41 2017 +0100

        KNOX-1035 - Move JWTTokenTest to gateway-spi module + add more tests

        Show
        dionusos Denes Bodo added a comment - $ git log -1 commit 773ac9d8e2594241d5d95448b19a75982f467efd Author: Colm O hEigeartaigh <coheigea@apache.org> Date: Mon Sep 11 10:29:41 2017 +0100 KNOX-1035 - Move JWTTokenTest to gateway-spi module + add more tests
        Hide
        coheigea Colm O hEigeartaigh added a comment -

        As a sanity test, this is still failing for me. Does it work for you?

        git clone https://git-wip-us.apache.org/repos/asf/knox.git
        cd knox
        wget https://issues.apache.org/jira/secure/attachment/12886354/0001-KNOX-1004-Failing-flaky-Knox-unit-test-org.apache.ha.patch
        git apply 0001-KNOX-1004-Failing-flaky-Knox-unit-test-org.apache.ha.patch

        Show
        coheigea Colm O hEigeartaigh added a comment - As a sanity test, this is still failing for me. Does it work for you? git clone https://git-wip-us.apache.org/repos/asf/knox.git cd knox wget https://issues.apache.org/jira/secure/attachment/12886354/0001-KNOX-1004-Failing-flaky-Knox-unit-test-org.apache.ha.patch git apply 0001- KNOX-1004 -Failing-flaky-Knox-unit-test-org.apache.ha.patch
        Hide
        dionusos Denes Bodo added a comment -

        Using the given steps I ran into the error. I recreated the changes in that new repository. Build and tests are OK locally.

        Show
        dionusos Denes Bodo added a comment - Using the given steps I ran into the error. I recreated the changes in that new repository. Build and tests are OK locally.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 5c95ab4e31ee547807e83e4b2d972b4d33da9eb2 in knox's branch refs/heads/master from Denes Bodo
        [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=5c95ab4 ]

        KNOX-1004 Failing (flaky) Knox unit tests

        Signed-off-by: Colm O hEigeartaigh <coheigea@apache.org>

        Show
        jira-bot ASF subversion and git services added a comment - Commit 5c95ab4e31ee547807e83e4b2d972b4d33da9eb2 in knox's branch refs/heads/master from Denes Bodo [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=5c95ab4 ] KNOX-1004 Failing (flaky) Knox unit tests Signed-off-by: Colm O hEigeartaigh <coheigea@apache.org>
        Hide
        coheigea Colm O hEigeartaigh added a comment -

        Thanks for the patch. What repository were you using as a matter of interest?

        Show
        coheigea Colm O hEigeartaigh added a comment - Thanks for the patch. What repository were you using as a matter of interest?
        Hide
        dionusos Denes Bodo added a comment -

        Thank you for your help!

        I checked out from https://github.com/apache/knox.git . I do not know the time I was cloned it.

        Show
        dionusos Denes Bodo added a comment - Thank you for your help! I checked out from https://github.com/apache/knox.git . I do not know the time I was cloned it.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 5c95ab4e31ee547807e83e4b2d972b4d33da9eb2 in knox's branch refs/heads/KNOX-998-Package_Restructuring from Denes Bodo
        [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=5c95ab4 ]

        KNOX-1004 Failing (flaky) Knox unit tests

        Signed-off-by: Colm O hEigeartaigh <coheigea@apache.org>

        Show
        jira-bot ASF subversion and git services added a comment - Commit 5c95ab4e31ee547807e83e4b2d972b4d33da9eb2 in knox's branch refs/heads/ KNOX-998 -Package_Restructuring from Denes Bodo [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=5c95ab4 ] KNOX-1004 Failing (flaky) Knox unit tests Signed-off-by: Colm O hEigeartaigh <coheigea@apache.org>

          People

          • Assignee:
            Unassigned
            Reporter:
            dionusos Denes Bodo
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development