Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
-
Component/s: test
-
Labels:None
-
Environment:
jenkins
Description
Found this issue on HADOOP-11149.
Tests run: 8, Failures: 0, Errors: 8, Skipped: 0, Time elapsed: 0.949 sec <<< FAILURE! - in org.apache.hadoop.fs.shell.TestCopyPreserveFlag
testDirectoryCpWithP(org.apache.hadoop.fs.shell.TestCopyPreserveFlag) Time elapsed: 0.616 sec <<< ERROR!
java.io.IOException: Mkdirs failed to create d0 (exists=false, cwd=/testptch/hadoop/hadoop-common-project/hadoop-common/target/test/data/2/testStat)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:449)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:913)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:894)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:856)
at org.apache.hadoop.fs.FileSystem.createNewFile(FileSystem.java:1150)
at org.apache.hadoop.fs.shell.TestCopyPreserveFlag.initialize(TestCopyPreserveFlag.java:72)
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Chris Nauroth thank you for taking a look. Currently, does one Jenkins server run on same machine or run tests in parallel? If so, we can switch off the parallel test execution or we can fix the test to handle the failure of mkdir.
Jenkins will execute a single pre-commit run at a time on each of its nodes. During the pre-commit run, Surefire will start multiple processes to run JUnit tests in parallel.
I'd prefer if we could move ahead with individual test fixes instead of turning off parallel execution. We get a big performance boost from the parallelization (~45 minutes to run tests in a single process down to ~15 minutes with parallel execution as I recall). Turning it off now would lower our bandwidth for executing pre-commit runs.
all the tests use the same test path. If a per method one were generated via a @Rule/TestMethod field to generate a method name, which would be used to build that final path, this problem would go away
all the tests use the same test path.
In theory, that should be OK. An @After method deletes the directory between each test run. Tests are parallelized at the suite level, but individual methods within a suite run serially, so they shouldn't collide with each other.
It's odd that the automatic mkdirs triggered from createNewFile is failing. This recursively creates all directories in the path, but without a guarantee of atomicity. If the previously created directories in the ancestry got deleted concurrently in the middle of this mkdirs, then that could cause it. Perhaps we have a situation where a concurrent test suite is being naughty and deleting the whole test.build.data structure, not isolating itself to that JUnit process's workspace (the "2" directory component shown in the description).
An interesting first step might be to make a troubleshooting patch that just catches the error and prints the directory diagnostics, similar to what MiniDFSCluster does. Then, we could submit a few Jenkins runs to try to repro. That might give us a firmer lead about root cause.
mvn test -Dtest=TestCopyPreserveFlag succeeds but mvn test -Dtest=TestLocalFSContract*,TestCopyPreserveFlag always fails on my environment.
These tests should use independent test dirs.
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 11s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
| +1 | mvninstall | 7m 1s | trunk passed |
| +1 | compile | 6m 30s | trunk passed with JDK v1.8.0_66 |
| +1 | compile | 7m 11s | trunk passed with JDK v1.7.0_91 |
| +1 | checkstyle | 0m 21s | trunk passed |
| +1 | mvnsite | 1m 7s | trunk passed |
| +1 | mvneclipse | 0m 15s | trunk passed |
| +1 | findbugs | 1m 38s | trunk passed |
| +1 | javadoc | 0m 56s | trunk passed with JDK v1.8.0_66 |
| +1 | javadoc | 1m 7s | trunk passed with JDK v1.7.0_91 |
| +1 | mvninstall | 0m 43s | the patch passed |
| +1 | compile | 6m 27s | the patch passed with JDK v1.8.0_66 |
| +1 | javac | 6m 27s | the patch passed |
| +1 | compile | 7m 11s | the patch passed with JDK v1.7.0_91 |
| +1 | javac | 7m 11s | the patch passed |
| +1 | checkstyle | 0m 22s | the patch passed |
| +1 | mvnsite | 1m 3s | the patch passed |
| +1 | mvneclipse | 0m 14s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 1m 53s | the patch passed |
| +1 | javadoc | 0m 56s | the patch passed with JDK v1.8.0_66 |
| +1 | javadoc | 1m 8s | the patch passed with JDK v1.7.0_91 |
| -1 | unit | 7m 40s | hadoop-common in the patch failed with JDK v1.8.0_66. |
| -1 | unit | 7m 45s | hadoop-common in the patch failed with JDK v1.7.0_91. |
| +1 | asflicense | 0m 23s | Patch does not generate ASF License warnings. |
| 63m 11s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_66 Failed junit tests | hadoop.fs.contract.rawlocal.TestRawLocalContractUnderlyingFileBehavior |
| JDK v1.7.0_91 Failed junit tests | hadoop.fs.contract.rawlocal.TestRawLocalContractUnderlyingFileBehavior |
This message was automatically generated.
Test failures appear to be related to the patch.
Sure. I attached 002. Thanks, Jason Lowe.
 +1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 15s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
| +1 | mvninstall | 6m 57s | trunk passed |
| +1 | compile | 5m 48s | trunk passed with JDK v1.8.0_72 |
| +1 | compile | 6m 41s | trunk passed with JDK v1.7.0_95 |
| +1 | checkstyle | 0m 21s | trunk passed |
| +1 | mvnsite | 1m 5s | trunk passed |
| +1 | mvneclipse | 0m 14s | trunk passed |
| +1 | findbugs | 1m 35s | trunk passed |
| +1 | javadoc | 0m 55s | trunk passed with JDK v1.8.0_72 |
| +1 | javadoc | 1m 5s | trunk passed with JDK v1.7.0_95 |
| +1 | mvninstall | 0m 43s | the patch passed |
| +1 | compile | 5m 38s | the patch passed with JDK v1.8.0_72 |
| +1 | javac | 5m 38s | the patch passed |
| +1 | compile | 6m 45s | the patch passed with JDK v1.7.0_95 |
| +1 | javac | 6m 45s | the patch passed |
| +1 | checkstyle | 0m 21s | the patch passed |
| +1 | mvnsite | 1m 0s | the patch passed |
| +1 | mvneclipse | 0m 15s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 1m 47s | the patch passed |
| +1 | javadoc | 0m 53s | the patch passed with JDK v1.8.0_72 |
| +1 | javadoc | 1m 3s | the patch passed with JDK v1.7.0_95 |
| +1 | unit | 7m 13s | hadoop-common in the patch passed with JDK v1.8.0_72. |
| +1 | unit | 7m 34s | hadoop-common in the patch passed with JDK v1.7.0_95. |
| +1 | asflicense | 0m 24s | Patch does not generate ASF License warnings. |
| 59m 43s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12787385/HADOOP-12589.002.patch |
| JIRA Issue | |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 65f353cd3afd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / aeb13ef |
| Default Java | 1.7.0_95 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 |
| findbugs | v3.0.0 |
| JDK v1.7.0_95 Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/8593/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Max memory used | 77MB |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/8593/console |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
LGTM, +1.
Thanks for updating the patch! Looks good except for one question: why is FileSystemTestHelper#getTestRootPath logic duplicated in LocalFSContract? It looks like the only difference is test/build/data vs. target/test/data as a fallback, default location, and I'm guessing only one of them is "correct." Should we just have LocalFSContract call FileSystemTestHelper#getTestRootPath directly?
Should we just have LocalFSContract call FileSystemTestHelper#getTestRootPath directly?
Yeah. I attached 003 which uses FileSystemTestHelper#getTestRootDir.
| +1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 11s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
| +1 | mvninstall | 8m 17s | trunk passed |
| +1 | compile | 8m 17s | trunk passed with JDK v1.8.0_66 |
| +1 | compile | 8m 26s | trunk passed with JDK v1.7.0_91 |
| +1 | checkstyle | 0m 27s | trunk passed |
| +1 | mvnsite | 1m 21s | trunk passed |
| +1 | mvneclipse | 0m 16s | trunk passed |
| +1 | findbugs | 1m 54s | trunk passed |
| +1 | javadoc | 1m 8s | trunk passed with JDK v1.8.0_66 |
| +1 | javadoc | 1m 15s | trunk passed with JDK v1.7.0_91 |
| +1 | mvninstall | 0m 51s | the patch passed |
| +1 | compile | 8m 18s | the patch passed with JDK v1.8.0_66 |
| +1 | javac | 8m 18s | the patch passed |
| +1 | compile | 8m 10s | the patch passed with JDK v1.7.0_91 |
| +1 | javac | 8m 10s | the patch passed |
| +1 | checkstyle | 0m 23s | the patch passed |
| +1 | mvnsite | 1m 11s | the patch passed |
| +1 | mvneclipse | 0m 14s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 2m 3s | the patch passed |
| +1 | javadoc | 1m 1s | the patch passed with JDK v1.8.0_66 |
| +1 | javadoc | 1m 15s | the patch passed with JDK v1.7.0_91 |
| +1 | unit | 9m 15s | hadoop-common in the patch passed with JDK v1.8.0_66. |
| +1 | unit | 9m 2s | hadoop-common in the patch passed with JDK v1.7.0_91. |
| +1 | asflicense | 0m 25s | Patch does not generate ASF License warnings. |
| 74m 51s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12787693/HADOOP-12589.003.patch |
| JIRA Issue | |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 336dfc2965c0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e6a7044 |
| Default Java | 1.7.0_91 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 |
| findbugs | v3.0.0 |
| JDK v1.7.0_91 Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/8611/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/8611/console |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
+1 latest patch lgtm.
+1 for the latest patch.
FAILURE: Integrated in Hadoop-trunk-Commit #9300 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9300/)
HADOOP-12589. Fix intermittent test failure of TestCopyPreserveFlag (iwasakims: rev 36ba5cac37dee6b60e607401ac06d263e4ac5d8f)
- hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/localfs/LocalFSContract.java
- hadoop-common-project/hadoop-common/CHANGES.txt
- hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java
Committed this to branch-2.6 and above. Thanks for the reviews!
Thanks for Masatake Iwasaki for your contribution and committing and thanks Jason Lowe and Akira Ajisaka for reviews!
Closing the JIRA as part of 2.7.3 release.
I'm pretty sure this has something to do with a resource conflict (probably local file system path) during parallel test execution. I don't recall ever seeing this fail before we turned on parallel test execution.