Hadoop Common
  1. Hadoop Common
  2. HADOOP-9051

“ant test” will build failed for trying to delete a file

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.4
    • Fix Version/s: 1.1.2
    • Component/s: test
    • Labels:
    • Environment:

      OS: Ubuntu 10.04; forrest version 0.8; findbugs version 2.0.1; ant version 1.8.1

    • Target Version/s:

      Description

      Run "ant test" on branch-1 of hadoop-common.
      When the test process reach "test-core-excluding-commit-and-smoke"

      It will invoke the "macro-test-runner" to clear and rebuild the test environment.
      Then the ant task command <delete dir="@

      {test.dir}

      /logs" />
      failed for trying to delete an non-existent file.

      following is the test result logs:
      test-core-excluding-commit-and-smoke:
      [delete] Deleting: /home/jdu/bdc/hadoop-topology-branch1-new/hadoop-common/build/test/testsfailed
      [delete] Deleting directory /home/jdu/bdc/hadoop-topology-branch1-new/hadoop-common/build/test/data
      [mkdir] Created dir: /home/jdu/bdc/hadoop-topology-branch1-new/hadoop-common/build/test/data
      [delete] Deleting directory /home/jdu/bdc/hadoop-topology-branch1-new/hadoop-common/build/test/logs

      BUILD FAILED
      /home/jdu/bdc/hadoop-topology-branch1-new/hadoop-common/build.xml:1212: The following error occurred while executing this line:
      /home/jdu/bdc/hadoop-topology-branch1-new/hadoop-common/build.xml:1166: The following error occurred while executing this line:
      /home/jdu/bdc/hadoop-topology-branch1-new/hadoop-common/build.xml:1057: Unable to delete file /home/jdu/bdc/hadoop-topology-branch1-new/hadoop-common/build/test/logs/userlogs/job_20121112223129603_0001/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/attempt_20121112223129603_0001_r_000000_0/stdout

      1. fix-ant-test
        0.7 kB
        meng gong
      2. hadoop-9051-v1.patch
        1.0 kB
        Luke Lu

        Activity

        Hide
        Matt Foley added a comment -

        Closed upon successful release of 1.1.2.

        Show
        Matt Foley added a comment - Closed upon successful release of 1.1.2.
        Hide
        Luke Lu added a comment -

        But please make sure a committer +1s the change.

        +1. Was in a hurry to unblock branch-1 testing. Should've let others finish the patch

        Show
        Luke Lu added a comment - But please make sure a committer +1s the change. +1. Was in a hurry to unblock branch-1 testing. Should've let others finish the patch
        Hide
        Suresh Srinivas added a comment -

        Luke, I know this is a trivial patch. But please make sure a committer +1s the change.

        Show
        Suresh Srinivas added a comment - Luke, I know this is a trivial patch. But please make sure a committer +1s the change.
        Hide
        Luke Lu added a comment -

        Committed trivial patch to branch-1* branches. Thanks for verifying the patch Junping.

        Show
        Luke Lu added a comment - Committed trivial patch to branch-1* branches. Thanks for verifying the patch Junping.
        Hide
        Junping Du added a comment -

        Hi Luke, Thanks for the patch. I test it on my local environment and do resolve the same problem. +1 on the patch.

        Show
        Junping Du added a comment - Hi Luke, Thanks for the patch. I test it on my local environment and do resolve the same problem. +1 on the patch.
        Hide
        Junping Du added a comment -

        Sorry. I didn't realize I reply a letter coming from jira. Just ignore above comments please.

        Show
        Junping Du added a comment - Sorry. I didn't realize I reply a letter coming from jira. Just ignore above comments please.
        Hide
        Luke Lu added a comment -

        Looks like the circular symlinks are caused by TestJobTrackerSafeMode and TestJobTrackerRestartWithLostTracker, which are both ignorable tests for now.

        Show
        Luke Lu added a comment - Looks like the circular symlinks are caused by TestJobTrackerSafeMode and TestJobTrackerRestartWithLostTracker, which are both ignorable tests for now.
        Hide
        Luke Lu added a comment -

        Note, this is for branch-1.* only (seems to affect branch-1 more than branch-1.0, as former creates more circular symlinks then the latter. branch-1.0 can actually finish with the same errors at the end of the tests). Maven build in trunk doesn't seem to suffer from the peculiar delete dir semantics.

        Show
        Luke Lu added a comment - Note, this is for branch-1.* only (seems to affect branch-1 more than branch-1.0, as former creates more circular symlinks then the latter. branch-1.0 can actually finish with the same errors at the end of the tests). Maven build in trunk doesn't seem to suffer from the peculiar delete dir semantics.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12560864/hadoop-9051-v1.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1867//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12560864/hadoop-9051-v1.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1867//console This message is automatically generated.
        Hide
        Luke Lu added a comment -

        It appears that some MR tests are creating circular symlinks of attempt dirs. The failure is due to ant delete dir by default following symlinks for "historical reasons" (cf. https://ant.apache.org/manual/Tasks/delete.html, which is different from the semantics of rm -rf that doesn't follow symlinks). Once the failure occurrs, the subsequent ant clean runs fail in a similar fashion.

        The hadoop-9051-v1.patch will make allow all tests to finish properly. Will file a separate JIRA to fix the offending tests.

        Show
        Luke Lu added a comment - It appears that some MR tests are creating circular symlinks of attempt dirs. The failure is due to ant delete dir by default following symlinks for "historical reasons" (cf. https://ant.apache.org/manual/Tasks/delete.html , which is different from the semantics of rm -rf that doesn't follow symlinks). Once the failure occurrs, the subsequent ant clean runs fail in a similar fashion. The hadoop-9051-v1.patch will make allow all tests to finish properly. Will file a separate JIRA to fix the offending tests.
        Hide
        Steve Loughran added a comment -

        Trying to delete a nonexistent directory is not the problem, otherwise ant clean wouldn't be idempotent. Something else is going wrong here.

        1. it may be windows only
        2. it may be some race condition with in-test cleanup taking place while directory cleanup takes place

        try doing a quick <sleep> before the delete to see if the error goes away. If so it is a race condition -which could be dodged either by setting that quiet=true -once we're confident it's a race condition.

        Show
        Steve Loughran added a comment - Trying to delete a nonexistent directory is not the problem, otherwise ant clean wouldn't be idempotent. Something else is going wrong here. it may be windows only it may be some race condition with in-test cleanup taking place while directory cleanup takes place try doing a quick <sleep> before the delete to see if the error goes away. If so it is a race condition -which could be dodged either by setting that quiet=true -once we're confident it's a race condition.
        Hide
        Junping Du added a comment -

        Hi Steve, What do you suggest here? I seems to run into the same issue before, but not sure how to fix that. It seems in this case, ant tries to delete a directory not exist (which put file (or sub-directory) names under a directory into a directory name). I just discussed with Meng, he is still in investigating but suspect ant has a bug from this behaviour. Any suggestion here?

        Show
        Junping Du added a comment - Hi Steve, What do you suggest here? I seems to run into the same issue before, but not sure how to fix that. It seems in this case, ant tries to delete a directory not exist (which put file (or sub-directory) names under a directory into a directory name). I just discussed with Meng, he is still in investigating but suspect ant has a bug from this behaviour. Any suggestion here?
        Hide
        Steve Loughran added a comment -

        Ant doesn't complain about deleting files/dirs that aren't there -otherwise "ant clean" wouldn't be idempotent.

        If it couldn't delete a file, then there's another cause -such as the file is in use.

        Setting quiet=true will keep the build going on after this -but the root cause is still lurking out there -maybe an MR process is still running.

        Is this replicable on your system? if you insert an <exec command="jps -v" /> , what processes are seen to be running?

        Show
        Steve Loughran added a comment - Ant doesn't complain about deleting files/dirs that aren't there -otherwise "ant clean" wouldn't be idempotent. If it couldn't delete a file, then there's another cause -such as the file is in use. Setting quiet=true will keep the build going on after this -but the root cause is still lurking out there -maybe an MR process is still running. Is this replicable on your system? if you insert an <exec command="jps -v" /> , what processes are seen to be running?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12553753/fix-ant-test
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1758//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12553753/fix-ant-test against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1758//console This message is automatically generated.
        Hide
        meng gong added a comment -

        an solution to fix this bug

        Show
        meng gong added a comment - an solution to fix this bug
        Hide
        meng gong added a comment -

        One kind of solution is following:
        — a/build.xml
        +++ b/build.xml
        @@ -1054,7 +1054,7 @@
        <delete file="$

        {test.build.dir}

        /testsfailed"/>
        <delete dir="@

        {test.dir}/data" />
        <mkdir dir="@{test.dir}

        /data" />

        • <delete dir="@ {test.dir}/logs" />
          + <delete dir="@{test.dir}

          /logs" quiet="true"/>
          <mkdir dir="@

          {test.dir}/logs" />
          <copy file="${test.src.dir}/hadoop-policy.xml"
          todir="@{test.dir}

          /extraconf" />

        For the marco-test-runner is just want to delete the logs folder and rebuild a new one.
        If the specified file or directory does not exist, do not display a diagnostic message.

        Show
        meng gong added a comment - One kind of solution is following: — a/build.xml +++ b/build.xml @@ -1054,7 +1054,7 @@ <delete file="$ {test.build.dir} /testsfailed"/> <delete dir="@ {test.dir}/data" /> <mkdir dir="@{test.dir} /data" /> <delete dir="@ {test.dir}/logs" /> + <delete dir="@{test.dir} /logs" quiet="true"/> <mkdir dir="@ {test.dir}/logs" /> <copy file="${test.src.dir}/hadoop-policy.xml" todir="@{test.dir} /extraconf" /> For the marco-test-runner is just want to delete the logs folder and rebuild a new one. If the specified file or directory does not exist, do not display a diagnostic message.

          People

          • Assignee:
            Luke Lu
            Reporter:
            meng gong
          • Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development