Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10722

Fix race condition in TestEditLog#testBatchedSyncWithClosedLogs

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: hdfs
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The test may fail the following assertion if async edit logs are enabled:
      logging edit without syncing should do not affect txid expected:<1> but was:<2>. The async thread is doing batched syncs in the background. logSync just ensures the edit is durable, so the txid may increase prior to sync. It's a race.

        Issue Links

          Activity

          Hide
          zhz Zhe Zhang added a comment -

          My bad, after pushing to branch-2.8 I found the JIRA was committed to branch-2.8 but didn't have the HDFS-7964 logic. I just did a revert and an Addendum. Sorry for messing up the git history.

          Show
          zhz Zhe Zhang added a comment - My bad, after pushing to branch-2.8 I found the JIRA was committed to branch-2.8 but didn't have the HDFS-7964 logic. I just did a revert and an Addendum. Sorry for messing up the git history.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for the work Daryn. I just backported to branch-2.8.

          Show
          zhz Zhe Zhang added a comment - Thanks for the work Daryn. I just backported to branch-2.8.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          As this is reverted from branch-2.8 ( async edit logging is not present in the branch-2.8), can we remove 2.8.0 from fix version ..?

          Show
          brahmareddy Brahma Reddy Battula added a comment - As this is reverted from branch-2.8 ( async edit logging is not present in the branch-2.8), can we remove 2.8.0 from fix version ..?
          Hide
          kihwal Kihwal Lee added a comment -

          Thanks for catching that, Arpit Agarwal. I try building/testing each branch before pushing, but must have forgotten it for branch-2.8.

          Show
          kihwal Kihwal Lee added a comment - Thanks for catching that, Arpit Agarwal . I try building/testing each branch before pushing, but must have forgotten it for branch-2.8.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10216 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10216/)
          HDFS-10722. Fix race condition in (kihwal: rev 4a8e6dc02fb1d0dc89c3981b444575494511444a)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10216 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10216/ ) HDFS-10722 . Fix race condition in (kihwal: rev 4a8e6dc02fb1d0dc89c3981b444575494511444a) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Looks like this commit broke the branch-2.8 build in tests. Pushed a change to fix the build, verified TestEditLog passes after this change.
          https://git-wip-us.apache.org/repos/asf?p=hadoop.git;a=commitdiff;h=f3424d662a662dd7148c5aa8bb59be2e558afa0c

          Show
          arpitagarwal Arpit Agarwal added a comment - Looks like this commit broke the branch-2.8 build in tests. Pushed a change to fix the build, verified TestEditLog passes after this change. https://git-wip-us.apache.org/repos/asf?p=hadoop.git;a=commitdiff;h=f3424d662a662dd7148c5aa8bb59be2e558afa0c
          Hide
          kihwal Kihwal Lee added a comment -

          Committed this to trunk, branch-2 and branch-2.8. Thanks for looking at this issue, Rakesh R.

          Show
          kihwal Kihwal Lee added a comment - Committed this to trunk, branch-2 and branch-2.8. Thanks for looking at this issue, Rakesh R .
          Hide
          kihwal Kihwal Lee added a comment -

          +1

          Show
          kihwal Kihwal Lee added a comment - +1
          Hide
          kihwal Kihwal Lee added a comment -

          Filed HDFS-10723 for TestRenameWhileOpen. HDFS-6532 is for TestCrcCorruption. TestTracing timed out. It passes when rerun locally.

          Show
          kihwal Kihwal Lee added a comment - Filed HDFS-10723 for TestRenameWhileOpen . HDFS-6532 is for TestCrcCorruption . TestTracing timed out. It passes when rerun locally.
          Hide
          daryn Daryn Sharp added a comment -

          Fixing a test doesn't cause 3 other tests to fail........

          Show
          daryn Daryn Sharp added a comment - Fixing a test doesn't cause 3 other tests to fail........
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 32s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 46s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 45s the patch passed
          +1 javac 0m 45s the patch passed
          +1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 139 unchanged - 1 fixed = 139 total (was 140)
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 50s the patch passed
          +1 javadoc 0m 52s the patch passed
          -1 unit 60m 4s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          80m 11s



          Reason Tests
          Failed junit tests hadoop.tracing.TestTracing
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.TestRenameWhileOpen



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822068/HDFS-10722.patch
          JIRA Issue HDFS-10722
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 77d58df3d989 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 70c2781
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16318/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16318/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16318/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 32s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 46s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 50s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed +1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 139 unchanged - 1 fixed = 139 total (was 140) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 50s the patch passed +1 javadoc 0m 52s the patch passed -1 unit 60m 4s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 80m 11s Reason Tests Failed junit tests hadoop.tracing.TestTracing   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.TestRenameWhileOpen Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822068/HDFS-10722.patch JIRA Issue HDFS-10722 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 77d58df3d989 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 70c2781 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16318/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16318/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16318/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          daryn Daryn Sharp added a comment -

          The remainder of the test case still has value so I wouldn't short out the entire test.

          Show
          daryn Daryn Sharp added a comment - The remainder of the test case still has value so I wouldn't short out the entire test.
          Hide
          rakeshr Rakesh R added a comment - - edited

          IMHO, to skip this test case something similar to TestEditLog_testSyncBatching approach rather than skipping only the assertion part, am I missing anything here?

          Show
          rakeshr Rakesh R added a comment - - edited IMHO, to skip this test case something similar to TestEditLog_testSyncBatching approach rather than skipping only the assertion part, am I missing anything here?
          Hide
          rakeshr Rakesh R added a comment -

          Daryn Sharp, there is already an issue created quite sometime back HDFS-10169 to handle this case. I had attached patch over there, appreciate reviews. Thanks!

          Show
          rakeshr Rakesh R added a comment - Daryn Sharp , there is already an issue created quite sometime back HDFS-10169 to handle this case. I had attached patch over there, appreciate reviews. Thanks!

            People

            • Assignee:
              daryn Daryn Sharp
              Reporter:
              daryn Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development