Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4717

TestResourceLocalizationService.testPublicResourceInitializesLocalDir fails Intermittently due to IllegalArgumentException from cleanup

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
    • Component/s: nodemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The same issue that was resolved by zhihai xu in YARN-3602 is back. Looks like the commons-io package throws an IAE instead of an IOE now if the directory doesn't exist.

      1. YARN-4717.001.patch
        1 kB
        Daniel Templeton

        Issue Links

          Activity

          Hide
          templedf Daniel Templeton added a comment -

          This is what the stack trace looks like:

          Running org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService
          Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.15 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService
          testPublicResourceInitializesLocalDir(org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService) Time elapsed: 1.71 sec <<< ERROR!
          java.lang.IllegalArgumentException: target/org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService/3/filecache/11 does not exist
          at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1637)
          at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
          at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
          at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
          at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
          at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
          at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
          at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
          at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
          at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
          at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
          at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService.cleanup(TestResourceLocalizationService.java:196)

          Results :

          Tests in error:
          TestResourceLocalizationService.cleanup:196 » IllegalArgument target/org.apach...

          Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

          Show
          templedf Daniel Templeton added a comment - This is what the stack trace looks like: Running org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.15 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService testPublicResourceInitializesLocalDir(org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService) Time elapsed: 1.71 sec <<< ERROR! java.lang.IllegalArgumentException: target/org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService/3/filecache/11 does not exist at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1637) at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535) at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270) at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653) at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535) at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270) at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653) at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535) at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270) at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653) at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService.cleanup(TestResourceLocalizationService.java:196) Results : Tests in error: TestResourceLocalizationService.cleanup:196 » IllegalArgument target/org.apach... Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
          Hide
          templedf Daniel Templeton added a comment -

          The exception happens when the recursive delete in the cleanup discovers that another thread has deleted one of the subdirectories it was planning to delete. This IAE is in addition to the IOE that can happen for the same reason.

          Show
          templedf Daniel Templeton added a comment - The exception happens when the recursive delete in the cleanup discovers that another thread has deleted one of the subdirectories it was planning to delete. This IAE is in addition to the IOE that can happen for the same reason.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 18s trunk passed
          +1 compile 0m 29s trunk passed with JDK v1.8.0_74
          +1 compile 0m 31s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 31s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 0s trunk passed
          +1 javadoc 0m 19s trunk passed with JDK v1.8.0_74
          +1 javadoc 0m 23s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 26s the patch passed
          +1 compile 0m 23s the patch passed with JDK v1.8.0_74
          +1 javac 0m 23s the patch passed
          +1 compile 0m 26s the patch passed with JDK v1.7.0_95
          +1 javac 0m 26s the patch passed
          +1 checkstyle 0m 13s the patch passed
          +1 mvnsite 0m 28s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 5s the patch passed
          +1 javadoc 0m 18s the patch passed with JDK v1.8.0_74
          +1 javadoc 0m 23s the patch passed with JDK v1.7.0_95
          +1 unit 9m 25s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74.
          +1 unit 9m 44s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          35m 42s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792588/YARN-4717.001.patch
          JIRA Issue YARN-4717
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 257b1a718436 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / adf1cdf
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10757/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10757/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 18s trunk passed +1 compile 0m 29s trunk passed with JDK v1.8.0_74 +1 compile 0m 31s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 31s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 0s trunk passed +1 javadoc 0m 19s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 23s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 26s the patch passed +1 compile 0m 23s the patch passed with JDK v1.8.0_74 +1 javac 0m 23s the patch passed +1 compile 0m 26s the patch passed with JDK v1.7.0_95 +1 javac 0m 26s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 28s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 5s the patch passed +1 javadoc 0m 18s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 23s the patch passed with JDK v1.7.0_95 +1 unit 9m 25s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74. +1 unit 9m 44s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 35m 42s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792588/YARN-4717.001.patch JIRA Issue YARN-4717 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 257b1a718436 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / adf1cdf Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10757/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10757/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          +1 LGTM easy fix

          Show
          rkanter Robert Kanter added a comment - +1 LGTM easy fix
          Hide
          rkanter Robert Kanter added a comment -

          Thanks Daniel. Committed to trunk and branch-2!

          Show
          rkanter Robert Kanter added a comment - Thanks Daniel. Committed to trunk and branch-2!
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Daniel Templeton / Robert Kanter, does this exist on previous branches too? If so, can this be backported to 2.8.0 / 2.7.x etc?

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Daniel Templeton / Robert Kanter , does this exist on previous branches too? If so, can this be backported to 2.8.0 / 2.7.x etc?
          Hide
          ebadger Eric Badger added a comment -

          Daniel Templeton, Robert Kanter, can we cherry-pick this back to 2.7? I just saw this failure in our nightly build.

          Show
          ebadger Eric Badger added a comment - Daniel Templeton , Robert Kanter , can we cherry-pick this back to 2.7? I just saw this failure in our nightly build.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks, Daniel Templeton! I committed this to branch-2.8 and branch-2.7 as well.

          Show
          jlowe Jason Lowe added a comment - Thanks, Daniel Templeton ! I committed this to branch-2.8 and branch-2.7 as well.
          Hide
          andrew.wang Andrew Wang added a comment -

          FYI for future git log greppers, this one is missing the JIRA #.

          Show
          andrew.wang Andrew Wang added a comment - FYI for future git log greppers, this one is missing the JIRA #.

            People

            • Assignee:
              templedf Daniel Templeton
              Reporter:
              templedf Daniel Templeton
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development