Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9467

Fix data race accessing writeLockHeldTimeStamp in FSNamesystem

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This is a followup of HDFS-9145.

      Actually the value of writeLockInterval should be captured within the lock. The current code has a race on writeLockHeldTimeStamp. Thanks to Jing Zhao for reporting this.

      1. HDFS-9467.000.patch
        1 kB
        Mingliang Liu
      2. HDFS-9467.001.patch
        1 kB
        Mingliang Liu

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #645 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/645/)
          HDFS-9467. Fix data race accessing writeLockHeldTimeStamp in (jing9: rev e556c35b0596700f9ec9d0a51cf5027259d531b5)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #645 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/645/ ) HDFS-9467 . Fix data race accessing writeLockHeldTimeStamp in (jing9: rev e556c35b0596700f9ec9d0a51cf5027259d531b5) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #736 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/736/)
          HDFS-9467. Fix data race accessing writeLockHeldTimeStamp in (jing9: rev e556c35b0596700f9ec9d0a51cf5027259d531b5)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #736 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/736/ ) HDFS-9467 . Fix data race accessing writeLockHeldTimeStamp in (jing9: rev e556c35b0596700f9ec9d0a51cf5027259d531b5) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2581 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2581/)
          HDFS-9467. Fix data race accessing writeLockHeldTimeStamp in (jing9: rev e556c35b0596700f9ec9d0a51cf5027259d531b5)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2581 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2581/ ) HDFS-9467 . Fix data race accessing writeLockHeldTimeStamp in (jing9: rev e556c35b0596700f9ec9d0a51cf5027259d531b5) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #725 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/725/)
          HDFS-9467. Fix data race accessing writeLockHeldTimeStamp in (jing9: rev e556c35b0596700f9ec9d0a51cf5027259d531b5)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #725 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/725/ ) HDFS-9467 . Fix data race accessing writeLockHeldTimeStamp in (jing9: rev e556c35b0596700f9ec9d0a51cf5027259d531b5) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2667 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2667/)
          HDFS-9467. Fix data race accessing writeLockHeldTimeStamp in (jing9: rev e556c35b0596700f9ec9d0a51cf5027259d531b5)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2667 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2667/ ) HDFS-9467 . Fix data race accessing writeLockHeldTimeStamp in (jing9: rev e556c35b0596700f9ec9d0a51cf5027259d531b5) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #1456 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1456/)
          HDFS-9467. Fix data race accessing writeLockHeldTimeStamp in (jing9: rev e556c35b0596700f9ec9d0a51cf5027259d531b5)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1456 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1456/ ) HDFS-9467 . Fix data race accessing writeLockHeldTimeStamp in (jing9: rev e556c35b0596700f9ec9d0a51cf5027259d531b5) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8893 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8893/)
          HDFS-9467. Fix data race accessing writeLockHeldTimeStamp in (jing9: rev e556c35b0596700f9ec9d0a51cf5027259d531b5)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8893 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8893/ ) HDFS-9467 . Fix data race accessing writeLockHeldTimeStamp in (jing9: rev e556c35b0596700f9ec9d0a51cf5027259d531b5) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          liuml07 Mingliang Liu added a comment -

          Thank you for reviewing this patch again and committing it.

          Show
          liuml07 Mingliang Liu added a comment - Thank you for reviewing this patch again and committing it.
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks for updating the patch, Mingliang Liu. Since the new change is trivial, I've committed this into trunk and branch-2 without running Jenkins again.

          Show
          jingzhao Jing Zhao added a comment - Thanks for updating the patch, Mingliang Liu . Since the new change is trivial, I've committed this into trunk and branch-2 without running Jenkins again.
          Hide
          liuml07 Mingliang Liu added a comment -

          The failing tests seem unrelated.

          Thanks Jing Zhao for your review. The v1 patch is to address the latest comment.

          Show
          liuml07 Mingliang Liu added a comment - The failing tests seem unrelated. Thanks Jing Zhao for your review. The v1 patch is to address the latest comment.
          Hide
          jingzhao Jing Zhao added a comment -

          The patch looks good to me. One nit is that the "isInfoEnabled" check may be unnecessary. Other than this +1.

          Show
          jingzhao Jing Zhao added a comment - The patch looks good to me. One nit is that the "isInfoEnabled" check may be unnecessary. Other than this +1.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 9m 21s trunk passed
          +1 compile 0m 54s trunk passed with JDK v1.8.0_66
          +1 compile 0m 51s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 2m 27s trunk passed
          +1 javadoc 1m 35s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 30s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 3s the patch passed
          +1 compile 1m 8s the patch passed with JDK v1.8.0_66
          +1 javac 1m 8s the patch passed
          +1 compile 0m 57s the patch passed with JDK v1.7.0_85
          +1 javac 0m 57s the patch passed
          -1 checkstyle 0m 20s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 248, now 248).
          +1 mvnsite 1m 9s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 38s the patch passed
          +1 javadoc 1m 33s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 22s the patch passed with JDK v1.7.0_85
          -1 unit 72m 25s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 68m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
          -1 asflicense 0m 34s Patch generated 56 ASF License warnings.
          175m 54s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.security.TestPermission
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.fs.viewfs.TestViewFsAtHdfsRoot
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.cli.TestHDFSCLI
          JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestAclsEndToEnd
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.TestErasureCodeBenchmarkThroughput
            hadoop.security.TestPermission
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774255/HDFS-9467.000.patch
          JIRA Issue HDFS-9467
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6b5b1c90fb1d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b4c6b51
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13656/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13656/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13656/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13656/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13656/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13656/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13656/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 78MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13656/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 9m 21s trunk passed +1 compile 0m 54s trunk passed with JDK v1.8.0_66 +1 compile 0m 51s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 19s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 2m 27s trunk passed +1 javadoc 1m 35s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 30s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 3s the patch passed +1 compile 1m 8s the patch passed with JDK v1.8.0_66 +1 javac 1m 8s the patch passed +1 compile 0m 57s the patch passed with JDK v1.7.0_85 +1 javac 0m 57s the patch passed -1 checkstyle 0m 20s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 248, now 248). +1 mvnsite 1m 9s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 38s the patch passed +1 javadoc 1m 33s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 22s the patch passed with JDK v1.7.0_85 -1 unit 72m 25s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 68m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 34s Patch generated 56 ASF License warnings. 175m 54s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.security.TestPermission   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.fs.viewfs.TestViewFsAtHdfsRoot JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.cli.TestHDFSCLI JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestAclsEndToEnd   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.TestErasureCodeBenchmarkThroughput   hadoop.security.TestPermission   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774255/HDFS-9467.000.patch JIRA Issue HDFS-9467 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6b5b1c90fb1d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b4c6b51 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13656/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13656/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13656/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13656/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13656/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13656/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13656/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 78MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13656/console This message was automatically generated.

            People

            • Assignee:
              liuml07 Mingliang Liu
              Reporter:
              liuml07 Mingliang Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development