Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12299

Race Between update pipeline and DN Re-Registration

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Scenario

      • Started pipeline with DN1->DN2->DN3
      • DN1 is re-reg and update pipeline is called
      • Update pipeline will success with DN1->DN3->DN4
      • Again update pipeline is called,which will fail with NPE.

      In step3 updatepipeline will set the storages as null since DN1 re-reg(which will remove and add the storages)
      FSNamesystem#updatePipelineInternal

         lastBlock.getUnderConstructionFeature().setExpectedLocations(lastBlock,
              storages, lastBlock.getBlockType())
      
      1. HDFS-12299.patch
        4 kB
        Brahma Reddy Battula
      2. HDFS-12299-branch-2.patch
        5 kB
        Brahma Reddy Battula
      3. HDFS-12299-branch-2-002.patch
        5 kB
        Brahma Reddy Battula

        Issue Links

          Activity

          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Kihwal Lee thanks for review and commit.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Kihwal Lee thanks for review and commit.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12243 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12243/)
          HDFS-12299. Race Between update pipeline and DN Re-Registration (kihwal: rev 8455d70756b584ddf27fc626a147f4eb2e1dc94e)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientProtocolForPipelineRecovery.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12243 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12243/ ) HDFS-12299 . Race Between update pipeline and DN Re-Registration (kihwal: rev 8455d70756b584ddf27fc626a147f4eb2e1dc94e) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientProtocolForPipelineRecovery.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          Hide
          kihwal Kihwal Lee added a comment -

          I've committed this to trunk, branch-2, branch-2.8 and branch-2.8.2. Thanks for working on this, Brahma.

          Show
          kihwal Kihwal Lee added a comment - I've committed this to trunk, branch-2, branch-2.8 and branch-2.8.2. Thanks for working on this, Brahma.
          Hide
          kihwal Kihwal Lee added a comment -

          +1 LGTM

          Show
          kihwal Kihwal Lee added a comment - +1 LGTM
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Updated patch to fix the checkstyle and testcase to reproduce all the time without fix.

          Kindly review.IMHO, this should go to 2.8.2 release.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Updated patch to fix the checkstyle and testcase to reproduce all the time without fix. Kindly review.IMHO, this should go to 2.8.2 release.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                branch-2 Compile Tests
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 6m 45s branch-2 passed
          +1 compile 1m 16s branch-2 passed with JDK v1.8.0_144
          +1 compile 1m 23s branch-2 passed with JDK v1.7.0_131
          +1 checkstyle 0m 34s branch-2 passed
          +1 mvnsite 1m 25s branch-2 passed
          +1 findbugs 3m 36s branch-2 passed
          +1 javadoc 0m 56s branch-2 passed with JDK v1.8.0_144
          +1 javadoc 1m 22s branch-2 passed with JDK v1.7.0_131
                Patch Compile Tests
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 12s the patch passed
          +1 compile 1m 13s the patch passed with JDK v1.8.0_144
          +1 javac 1m 13s the patch passed
          +1 compile 1m 21s the patch passed with JDK v1.7.0_131
          +1 javac 1m 21s the patch passed
          -0 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 1 new + 80 unchanged - 0 fixed = 81 total (was 80)
          +1 mvnsite 1m 21s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 59s the patch passed
          +1 javadoc 0m 53s the patch passed with JDK v1.8.0_144
          +1 javadoc 1m 22s the patch passed with JDK v1.7.0_131
                Other Tests
          +1 unit 1m 13s hadoop-hdfs-client in the patch passed with JDK v1.7.0_131.
          -1 unit 61m 23s hadoop-hdfs in the patch failed with JDK v1.7.0_131.
          -1 asflicense 0m 29s The patch generated 1 ASF License warnings.
          160m 24s



          Reason Tests
          JDK v1.8.0_144 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            org.apache.hadoop.hdfs.server.namenode.ha.TestHAMetrics
          JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain
            hadoop.hdfs.server.balancer.TestBalancerRPCDelay
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs
          JDK v1.7.0_131 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:5e40efe
          JIRA Issue HDFS-12299
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882851/HDFS-12299-branch-2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 733b89f3b712 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / d5379cc
          Default Java 1.7.0_131
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20780/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/20780/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt
          JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20780/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/20780/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20780/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2 Compile Tests 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 6m 45s branch-2 passed +1 compile 1m 16s branch-2 passed with JDK v1.8.0_144 +1 compile 1m 23s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 34s branch-2 passed +1 mvnsite 1m 25s branch-2 passed +1 findbugs 3m 36s branch-2 passed +1 javadoc 0m 56s branch-2 passed with JDK v1.8.0_144 +1 javadoc 1m 22s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 12s the patch passed +1 compile 1m 13s the patch passed with JDK v1.8.0_144 +1 javac 1m 13s the patch passed +1 compile 1m 21s the patch passed with JDK v1.7.0_131 +1 javac 1m 21s the patch passed -0 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 1 new + 80 unchanged - 0 fixed = 81 total (was 80) +1 mvnsite 1m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 59s the patch passed +1 javadoc 0m 53s the patch passed with JDK v1.8.0_144 +1 javadoc 1m 22s the patch passed with JDK v1.7.0_131       Other Tests +1 unit 1m 13s hadoop-hdfs-client in the patch passed with JDK v1.7.0_131. -1 unit 61m 23s hadoop-hdfs in the patch failed with JDK v1.7.0_131. -1 asflicense 0m 29s The patch generated 1 ASF License warnings. 160m 24s Reason Tests JDK v1.8.0_144 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   org.apache.hadoop.hdfs.server.namenode.ha.TestHAMetrics JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain   hadoop.hdfs.server.balancer.TestBalancerRPCDelay   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs JDK v1.7.0_131 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue HDFS-12299 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882851/HDFS-12299-branch-2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 733b89f3b712 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / d5379cc Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20780/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20780/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20780/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/20780/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20780/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment - - edited

          Uploading the test to reproduce this issue.

          HDFS-9040 fixes this issue fully in trunk, So only testcase is added.
          HDFS-11817 Partially (null storages were included) fixes in branch-2 and branch-2.8. Updated patch to add only non-null DatanodeStorageInfo at BlockUnderConstructionFeature#setExpectedLocations

          Kindly Review.

          Show
          brahmareddy Brahma Reddy Battula added a comment - - edited Uploading the test to reproduce this issue. HDFS-9040 fixes this issue fully in trunk , So only testcase is added. HDFS-11817 Partially (null storages were included) fixes in branch-2 and branch-2.8 . Updated patch to add only non-null DatanodeStorageInfo at BlockUnderConstructionFeature#setExpectedLocations Kindly Review.

            People

            • Assignee:
              brahmareddy Brahma Reddy Battula
              Reporter:
              brahmareddy Brahma Reddy Battula
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development