Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.8.0, 3.0.0-alpha1
-
Component/s: hdfs
-
Labels:None
-
Target Version/s:
-
Hadoop Flags:Reviewed
Description
The test may fail the following assertion if async edit logs are enabled:
logging edit without syncing should do not affect txid expected:<1> but was:<2>. The async thread is doing batched syncs in the background. logSync just ensures the edit is durable, so the txid may increase prior to sync. It's a race.
Issue Links
- is duplicated by
-
HDFS-10169
TestEditLog.testBatchedSyncWithClosedLogs with useAsyncEditLog sometimes fails
-
- Resolved
-
- relates to
-
HDFS-7964
Add support for async edit logging
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
IMHO, to skip this test case something similar to TestEditLog_testSyncBatching approach rather than skipping only the assertion part, am I missing anything here?
The remainder of the test case still has value so I wouldn't short out the entire test.
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 9s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 7m 32s | trunk passed |
| +1 | compile | 0m 46s | trunk passed |
| +1 | checkstyle | 0m 28s | trunk passed |
| +1 | mvnsite | 0m 55s | trunk passed |
| +1 | mvneclipse | 0m 13s | trunk passed |
| +1 | findbugs | 1m 46s | trunk passed |
| +1 | javadoc | 0m 55s | trunk passed |
| +1 | mvninstall | 0m 50s | the patch passed |
| +1 | compile | 0m 45s | the patch passed |
| +1 | javac | 0m 45s | the patch passed |
| +1 | checkstyle | 0m 26s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 139 unchanged - 1 fixed = 139 total (was 140) |
| +1 | mvnsite | 0m 53s | the patch passed |
| +1 | mvneclipse | 0m 10s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 50s | the patch passed |
| +1 | javadoc | 0m 52s | the patch passed |
| -1 | unit | 60m 4s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 21s | The patch does not generate ASF License warnings. |
| 80m 11s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.tracing.TestTracing |
| hadoop.hdfs.TestCrcCorruption | |
| hadoop.hdfs.TestRenameWhileOpen |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12822068/HDFS-10722.patch |
| JIRA Issue | |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 77d58df3d989 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 70c2781 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/16318/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16318/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16318/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Fixing a test doesn't cause 3 other tests to fail........
Filed HDFS-10723 for TestRenameWhileOpen. HDFS-6532 is for TestCrcCorruption. TestTracing timed out. It passes when rerun locally.
+1
Committed this to trunk, branch-2 and branch-2.8. Thanks for looking at this issue, Rakesh R.
Looks like this commit broke the branch-2.8 build in tests. Pushed a change to fix the build, verified TestEditLog passes after this change.
https://git-wip-us.apache.org/repos/asf?p=hadoop.git;a=commitdiff;h=f3424d662a662dd7148c5aa8bb59be2e558afa0c
SUCCESS: Integrated in Hadoop-trunk-Commit #10216 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10216/)
HDFS-10722. Fix race condition in (kihwal: rev 4a8e6dc02fb1d0dc89c3981b444575494511444a)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
Thanks for catching that, Arpit Agarwal. I try building/testing each branch before pushing, but must have forgotten it for branch-2.8.
As this is reverted from branch-2.8 ( async edit logging is not present in the branch-2.8), can we remove 2.8.0 from fix version ..?
Thanks for the work Daryn. I just backported to branch-2.8.
My bad, after pushing to branch-2.8 I found the JIRA was committed to branch-2.8 but didn't have the HDFS-7964 logic. I just did a revert and an Addendum. Sorry for messing up the git history.
Daryn Sharp, there is already an issue created quite sometime back
HDFS-10169to handle this case. I had attached patch over there, appreciate reviews. Thanks!