Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11094

Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: datanode
    • Labels:
      None

      Description

      The datanode should know which NN is active when it is connecting/registering to the NN. Currently, it only figures this out during its first (and subsequent) heartbeat(s) and so there is a period of time where the datanode is alive and registered, but can't actually do anything because it doesn't know which NN is active. A byproduct of this is that the MiniDFSCluster will become active before it knows what NN is active, which can lead to NPEs when calling getActiveNN().

      1. HDFS-11094.001.patch
        0.9 kB
        Eric Badger
      2. HDFS-11094.002.patch
        11 kB
        Eric Badger
      3. HDFS-11094.003.patch
        12 kB
        Eric Badger
      4. HDFS-11094.004.patch
        12 kB
        Eric Badger
      5. HDFS-11094.005.patch
        9 kB
        Eric Badger
      6. HDFS-11094.006.patch
        11 kB
        Eric Badger
      7. HDFS-11094.007.patch
        11 kB
        Eric Badger
      8. HDFS-11094.008.patch
        15 kB
        Eric Badger
      9. HDFS-11094.009.patch
        15 kB
        Eric Badger
      10. HDFS-11094.009-b2.patch
        18 kB
        Eric Badger
      11. HDFS-11094.010.patch
        17 kB
        Eric Badger
      12. HDFS-11094.010-b2.patch
        17 kB
        Eric Badger
      13. HDFS-11094.011.patch
        17 kB
        Eric Badger
      14. HDFS-11094-branch-2.011.patch
        18 kB
        Eric Badger

        Activity

        Hide
        ebadger Eric Badger added a comment -

        The problem is that the MiniDFSCluster doesn't wait for the DN(s) to heartbeat before it declares itself as "active" or "up". Since the DNs haven't heartbeated in yet, if the test asks the DN for the active NN it will return null. This will then cause the test to fail.

        So I am proposing that we wait for the DN(s) to hearbeat into the NN before allowing the cluster to return from its build() method. This may also have a performance boost since it will cause the DNs to be proactive in talking to the namenode instead of waiting and doing it on a timer.

        Show
        ebadger Eric Badger added a comment - The problem is that the MiniDFSCluster doesn't wait for the DN(s) to heartbeat before it declares itself as "active" or "up". Since the DNs haven't heartbeated in yet, if the test asks the DN for the active NN it will return null. This will then cause the test to fail. So I am proposing that we wait for the DN(s) to hearbeat into the NN before allowing the cluster to return from its build() method. This may also have a performance boost since it will cause the DNs to be proactive in talking to the namenode instead of waiting and doing it on a timer.
        Hide
        liuml07 Mingliang Liu added a comment -
        1. Will this suppress the heartbeat interval config, and fail other tests?
        2. As an alternative, can we simply change the unit test?
          FWIW, heartbeat is checked asynchronously in HeartbeatManager. Do we need to trigger the heartbeat check logic in NN?
              final BlockManager bm = cluster.getNamesystem().getBlockManager();
              // trigger NN handel heartbeat
              BlockManagerTestUtil.checkHeartbeat(bm);
          
        Show
        liuml07 Mingliang Liu added a comment - Will this suppress the heartbeat interval config, and fail other tests? As an alternative, can we simply change the unit test? FWIW, heartbeat is checked asynchronously in HeartbeatManager . Do we need to trigger the heartbeat check logic in NN? final BlockManager bm = cluster.getNamesystem().getBlockManager(); // trigger NN handel heartbeat BlockManagerTestUtil.checkHeartbeat(bm);
        Hide
        ebadger Eric Badger added a comment -

        Will this suppress the heartbeat interval config, and fail other tests?

        Yes, it definitely could. I forgot to submit the patch earlier, but I just did. We'll see how bad the damage is. I think it makes more sense to have the cluster not become active until it is actually fully usable.

        As an alternative, can we simply change the unit test?

        Yes, I've also tested another patch that uses triggerHeartbeatForTests() before calling getActiveNN() and it works. I would prefer to put it into the MiniDFSCluster, but if we can't do that then this will be good enough to fix this specific test.

        Show
        ebadger Eric Badger added a comment - Will this suppress the heartbeat interval config, and fail other tests? Yes, it definitely could. I forgot to submit the patch earlier, but I just did. We'll see how bad the damage is. I think it makes more sense to have the cluster not become active until it is actually fully usable. As an alternative, can we simply change the unit test? Yes, I've also tested another patch that uses triggerHeartbeatForTests() before calling getActiveNN() and it works. I would prefer to put it into the MiniDFSCluster, but if we can't do that then this will be good enough to fix this specific test.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 10s trunk passed
        +1 compile 0m 48s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 0m 53s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 47s trunk passed
        +1 javadoc 0m 39s trunk passed
        +1 mvninstall 0m 49s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        +1 checkstyle 0m 25s the patch passed
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 55s the patch passed
        +1 javadoc 0m 38s the patch passed
        -1 unit 58m 13s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        77m 34s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockReportRateLimiting
          hadoop.net.TestNetworkTopology



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-11094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836655/HDFS-11094.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 90d317eaecb6 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f2800f5
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17394/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17394/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17394/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 10s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 47s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 55s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 58m 13s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 77m 34s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockReportRateLimiting   hadoop.net.TestNetworkTopology Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-11094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836655/HDFS-11094.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 90d317eaecb6 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f2800f5 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17394/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17394/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17394/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ebadger Eric Badger added a comment -

        The tests that failed are related to this patch. They depend on the datanode not being able to heartbeat correctly, so the test will timeout when this fix is put in. Additionally, upon further review it looks like the cluster is actually waiting for the heartbeat to be received. The race is between the test checking for the active NN and the HAState getting set after a heartbeat response is received by the data node. After talking with Daryn Sharp, I think that allowing the NN to send back its HA state during registration is the best way to fix this. This way the datanodes will always know the HAState of the NN that they are connected to.

        I'm attaching a patch that adds the HAServiceState of the NN to the NamespaceInfo that gets sent from the NN to the DN during a versionRequest. This is one of the first things that happens during a DN-NN connection setup and happens even before registration of the DN.

        Daryn Sharp, Mingliang Liu, can you please review? Thanks

        Show
        ebadger Eric Badger added a comment - The tests that failed are related to this patch. They depend on the datanode not being able to heartbeat correctly, so the test will timeout when this fix is put in. Additionally, upon further review it looks like the cluster is actually waiting for the heartbeat to be received. The race is between the test checking for the active NN and the HAState getting set after a heartbeat response is received by the data node. After talking with Daryn Sharp , I think that allowing the NN to send back its HA state during registration is the best way to fix this. This way the datanodes will always know the HAState of the NN that they are connected to. I'm attaching a patch that adds the HAServiceState of the NN to the NamespaceInfo that gets sent from the NN to the DN during a versionRequest. This is one of the first things that happens during a DN-NN connection setup and happens even before registration of the DN. Daryn Sharp , Mingliang Liu , can you please review? Thanks
        Hide
        ebadger Eric Badger added a comment -

        Oops, accidentally uploaded a patch against 2.8, will rebase and upload again.

        Show
        ebadger Eric Badger added a comment - Oops, accidentally uploaded a patch against 2.8, will rebase and upload again.
        Hide
        ebadger Eric Badger added a comment -

        Rebased to trunk. Resubmitting patch

        Show
        ebadger Eric Badger added a comment - Rebased to trunk. Resubmitting patch
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 8m 43s trunk passed
        +1 compile 1m 42s trunk passed
        +1 checkstyle 0m 35s trunk passed
        +1 mvnsite 1m 32s trunk passed
        +1 mvneclipse 0m 24s trunk passed
        +1 findbugs 3m 15s trunk passed
        +1 javadoc 1m 0s trunk passed
        0 mvndep 0m 7s Maven dependency ordering for patch
        +1 mvninstall 1m 19s the patch passed
        +1 compile 1m 17s the patch passed
        +1 cc 1m 17s the patch passed
        +1 javac 1m 17s the patch passed
        -0 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 13 new + 256 unchanged - 7 fixed = 269 total (was 263)
        +1 mvnsite 1m 24s the patch passed
        +1 mvneclipse 0m 20s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 16s the patch passed
        +1 javadoc 0m 53s the patch passed
        +1 unit 0m 52s hadoop-hdfs-client in the patch passed.
        -1 unit 53m 47s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        83m 2s



        Reason Tests
        Failed junit tests hadoop.hdfs.qjournal.client.TestEpochsAreUnique
          hadoop.hdfs.qjournal.TestNNWithQJM
          hadoop.hdfs.TestSafeModeWithStripedFile
          hadoop.hdfs.qjournal.TestSecureNNWithQJM
          hadoop.hdfs.qjournal.client.TestQuorumJournalManager
          hadoop.hdfs.TestRollingUpgradeDowngrade
          hadoop.hdfs.qjournal.server.TestJournalNode
          hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM
          hadoop.hdfs.qjournal.client.TestQJMWithFaults
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
          hadoop.hdfs.tools.TestDFSAdminWithHA
          hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
          hadoop.hdfs.server.namenode.ha.TestStandbyInProgressTail
          hadoop.hdfs.TestRollingUpgradeRollback
          hadoop.hdfs.TestRollingUpgrade
          hadoop.hdfs.TestDFSInotifyEventInputStream
          hadoop.hdfs.protocolPB.TestPBHelper



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:e809691
        JIRA Issue HDFS-11094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838424/HDFS-11094.003.patch
        Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle
        uname Linux b0e6cba8cde9 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 89354f0
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17514/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17514/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17514/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17514/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 8m 43s trunk passed +1 compile 1m 42s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 32s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 3m 15s trunk passed +1 javadoc 1m 0s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 1m 17s the patch passed +1 cc 1m 17s the patch passed +1 javac 1m 17s the patch passed -0 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 13 new + 256 unchanged - 7 fixed = 269 total (was 263) +1 mvnsite 1m 24s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 16s the patch passed +1 javadoc 0m 53s the patch passed +1 unit 0m 52s hadoop-hdfs-client in the patch passed. -1 unit 53m 47s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 83m 2s Reason Tests Failed junit tests hadoop.hdfs.qjournal.client.TestEpochsAreUnique   hadoop.hdfs.qjournal.TestNNWithQJM   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.qjournal.client.TestQuorumJournalManager   hadoop.hdfs.TestRollingUpgradeDowngrade   hadoop.hdfs.qjournal.server.TestJournalNode   hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM   hadoop.hdfs.qjournal.client.TestQJMWithFaults   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.server.namenode.ha.TestStandbyInProgressTail   hadoop.hdfs.TestRollingUpgradeRollback   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.TestDFSInotifyEventInputStream   hadoop.hdfs.protocolPB.TestPBHelper Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HDFS-11094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838424/HDFS-11094.003.patch Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle uname Linux b0e6cba8cde9 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 89354f0 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17514/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17514/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17514/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17514/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ebadger Eric Badger added a comment -

        New patch checks for null before calling setState() to prevent the NPE present in all of the failing unit tests.

        Show
        ebadger Eric Badger added a comment - New patch checks for null before calling setState() to prevent the NPE present in all of the failing unit tests.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 4m 29s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 7s Maven dependency ordering for branch
        +1 mvninstall 6m 42s trunk passed
        +1 compile 1m 19s trunk passed
        +1 checkstyle 0m 33s trunk passed
        +1 mvnsite 1m 23s trunk passed
        +1 mvneclipse 0m 24s trunk passed
        +1 findbugs 3m 3s trunk passed
        +1 javadoc 0m 58s trunk passed
        0 mvndep 0m 6s Maven dependency ordering for patch
        +1 mvninstall 1m 14s the patch passed
        +1 compile 1m 17s the patch passed
        +1 cc 1m 17s the patch passed
        +1 javac 1m 17s the patch passed
        -0 checkstyle 0m 29s hadoop-hdfs-project: The patch generated 15 new + 256 unchanged - 7 fixed = 271 total (was 263)
        +1 mvnsite 1m 18s the patch passed
        +1 mvneclipse 0m 19s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 15s the patch passed
        +1 javadoc 0m 55s the patch passed
        +1 unit 0m 52s hadoop-hdfs-client in the patch passed.
        -1 unit 53m 49s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        84m 8s



        Reason Tests
        Failed junit tests hadoop.hdfs.tools.TestDelegationTokenFetcher



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838619/HDFS-11094.004.patch
        Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle
        uname Linux 254f09d01d65 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ede1a47
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17534/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17534/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17534/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17534/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 4m 29s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 6m 42s trunk passed +1 compile 1m 19s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 23s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 3m 3s trunk passed +1 javadoc 0m 58s trunk passed 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 1m 14s the patch passed +1 compile 1m 17s the patch passed +1 cc 1m 17s the patch passed +1 javac 1m 17s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project: The patch generated 15 new + 256 unchanged - 7 fixed = 271 total (was 263) +1 mvnsite 1m 18s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 15s the patch passed +1 javadoc 0m 55s the patch passed +1 unit 0m 52s hadoop-hdfs-client in the patch passed. -1 unit 53m 49s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 84m 8s Reason Tests Failed junit tests hadoop.hdfs.tools.TestDelegationTokenFetcher Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838619/HDFS-11094.004.patch Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle uname Linux 254f09d01d65 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ede1a47 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17534/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17534/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17534/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17534/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment -

        I think this should work. Having the HAState namespace info in the first part of the handshake with the NameNode makes sense. I +1 on this idea, but need 2nd opinion. Ping Daryn Sharp and Arpit Agarwal.

        One concern is about backward compatibility if we change HeartbeatRequestProto. By the way, can you update the subject and description of this JIRA to make it more dictative? The failing test is just one of the motivations now.

        Show
        liuml07 Mingliang Liu added a comment - I think this should work. Having the HAState namespace info in the first part of the handshake with the NameNode makes sense. I +1 on this idea, but need 2nd opinion. Ping Daryn Sharp and Arpit Agarwal . One concern is about backward compatibility if we change HeartbeatRequestProto . By the way, can you update the subject and description of this JIRA to make it more dictative? The failing test is just one of the motivations now.
        Hide
        ebadger Eric Badger added a comment -

        Mingliang Liu, I updated the subject and description.

        One concern is about backward compatibility if we change HeartbeatRequestProto.

        I don't think HeartbeatRequestProto actually needs to be changed. I was trying to normalize some of the message types that were being used, but made the normalization from the side of what I was adding instead of what was there. I'll fix this and upload a new patch.

        Show
        ebadger Eric Badger added a comment - Mingliang Liu , I updated the subject and description. One concern is about backward compatibility if we change HeartbeatRequestProto. I don't think HeartbeatRequestProto actually needs to be changed. I was trying to normalize some of the message types that were being used, but made the normalization from the side of what I was adding instead of what was there. I'll fix this and upload a new patch.
        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks for updating the JIRA. I changed the target version branch-2.9+

        Show
        liuml07 Mingliang Liu added a comment - Thanks for updating the JIRA. I changed the target version branch-2.9+
        Hide
        ebadger Eric Badger added a comment -

        Uploading new patch that doesn't touch NNHAStatusHeartbeat. Simplifies the patch and decreases the overall size.

        Show
        ebadger Eric Badger added a comment - Uploading new patch that doesn't touch NNHAStatusHeartbeat. Simplifies the patch and decreases the overall size.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 23s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 27s trunk passed
        +1 compile 0m 52s trunk passed
        +1 checkstyle 0m 33s trunk passed
        +1 mvnsite 0m 57s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 42s trunk passed
        +1 javadoc 0m 41s trunk passed
        +1 mvninstall 0m 51s the patch passed
        +1 compile 0m 46s the patch passed
        +1 cc 0m 46s the patch passed
        +1 javac 0m 46s the patch passed
        -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 24 new + 264 unchanged - 0 fixed = 288 total (was 264)
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 50s the patch passed
        +1 javadoc 0m 35s the patch passed
        -1 unit 76m 23s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 26s The patch does not generate ASF License warnings.
        96m 29s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM
          hadoop.hdfs.TestDFSInotifyEventInputStream
          hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR
          hadoop.hdfs.server.namenode.ha.TestStandbyInProgressTail
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
          hadoop.hdfs.TestRollingUpgrade
          hadoop.hdfs.qjournal.client.TestEpochsAreUnique
          hadoop.hdfs.protocolPB.TestPBHelper
          hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
          hadoop.hdfs.tools.TestDFSAdminWithHA
          hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
          hadoop.hdfs.qjournal.client.TestQuorumJournalManager
          hadoop.hdfs.qjournal.TestSecureNNWithQJM
          hadoop.hdfs.TestRollingUpgradeRollback
          hadoop.hdfs.qjournal.server.TestJournalNode
          hadoop.hdfs.qjournal.client.TestQJMWithFaults
          hadoop.hdfs.qjournal.TestNNWithQJM
          hadoop.hdfs.TestRollingUpgradeDowngrade



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839002/HDFS-11094.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 5e8264ac4141 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3219b7b
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17573/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17573/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17573/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17573/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 27s trunk passed +1 compile 0m 52s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 51s the patch passed +1 compile 0m 46s the patch passed +1 cc 0m 46s the patch passed +1 javac 0m 46s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 24 new + 264 unchanged - 0 fixed = 288 total (was 264) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 50s the patch passed +1 javadoc 0m 35s the patch passed -1 unit 76m 23s hadoop-hdfs in the patch failed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 96m 29s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM   hadoop.hdfs.TestDFSInotifyEventInputStream   hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR   hadoop.hdfs.server.namenode.ha.TestStandbyInProgressTail   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.qjournal.client.TestEpochsAreUnique   hadoop.hdfs.protocolPB.TestPBHelper   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.qjournal.client.TestQuorumJournalManager   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.TestRollingUpgradeRollback   hadoop.hdfs.qjournal.server.TestJournalNode   hadoop.hdfs.qjournal.client.TestQJMWithFaults   hadoop.hdfs.qjournal.TestNNWithQJM   hadoop.hdfs.TestRollingUpgradeDowngrade Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839002/HDFS-11094.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 5e8264ac4141 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3219b7b Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17573/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17573/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17573/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17573/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ebadger Eric Badger added a comment -

        New patch adds in INITIALIZING state to convert() methods to fix test failures. Optimized redundant code in convert() methods.

        Show
        ebadger Eric Badger added a comment - New patch adds in INITIALIZING state to convert() methods to fix test failures. Optimized redundant code in convert() methods.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 4m 23s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 3s trunk passed
        +1 compile 0m 44s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 40s trunk passed
        +1 javadoc 0m 37s trunk passed
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 42s the patch passed
        +1 cc 0m 42s the patch passed
        +1 javac 0m 42s the patch passed
        -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 27 new + 255 unchanged - 9 fixed = 282 total (was 264)
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 46s the patch passed
        +1 javadoc 0m 36s the patch passed
        -1 unit 56m 42s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        79m 26s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
          hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839065/HDFS-11094.006.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux e462c95850ca 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f121d0b
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17576/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17576/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17576/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17576/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 4m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 3s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 37s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 42s the patch passed +1 cc 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 27 new + 255 unchanged - 9 fixed = 282 total (was 264) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 36s the patch passed -1 unit 56m 42s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 79m 26s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839065/HDFS-11094.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux e462c95850ca 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f121d0b Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17576/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17576/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17576/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17576/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ebadger Eric Badger added a comment -

        The test failures are unrelated to the patch and do not fail for me locally.

        Mingliang Liu, Daryn Sharp, Arpit Agarwal, could you please review the latest patch? Thanks

        Show
        ebadger Eric Badger added a comment - The test failures are unrelated to the patch and do not fail for me locally. Mingliang Liu , Daryn Sharp , Arpit Agarwal , could you please review the latest patch? Thanks
        Hide
        liuml07 Mingliang Liu added a comment -

        +1

        Some comments:

        1. Is it possible to add a unit test for the HAState is sent during a versionRequest RPC?
        2. Add some logging will be very helpful, e.g. LOG.info("Acknowledging ACTIVE Namenode during handshake" + actor);
          313	    if(nsInfo.getState() == HAServiceState.ACTIVE && bpServiceToActive == null) {
          314	      bpServiceToActive = actor;
          315	    }
          
        3. The checkstyle warnings are related. Let's address them.

        Daryn Sharp and Arpit Agarwal If you don't have any comments, I'll commit this after the above are addressed.

        Show
        liuml07 Mingliang Liu added a comment - +1 Some comments: Is it possible to add a unit test for the HAState is sent during a versionRequest RPC? Add some logging will be very helpful, e.g. LOG.info("Acknowledging ACTIVE Namenode during handshake" + actor); 313 if (nsInfo.getState() == HAServiceState.ACTIVE && bpServiceToActive == null ) { 314 bpServiceToActive = actor; 315 } The checkstyle warnings are related. Let's address them. Daryn Sharp and Arpit Agarwal If you don't have any comments, I'll commit this after the above are addressed.
        Hide
        ebadger Eric Badger added a comment -

        I'm addressing the logging and checkstyle warnings in this patch. However, I will need some time to figure out how to do the unit testing. Mingliang Liu, do you have any suggestions? It seems like this will be quite difficult to mock out.

        Show
        ebadger Eric Badger added a comment - I'm addressing the logging and checkstyle warnings in this patch. However, I will need some time to figure out how to do the unit testing. Mingliang Liu , do you have any suggestions? It seems like this will be quite difficult to mock out.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 28s trunk passed
        +1 compile 0m 56s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 1m 3s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 52s trunk passed
        +1 javadoc 0m 44s trunk passed
        +1 mvninstall 0m 47s the patch passed
        +1 compile 0m 44s the patch passed
        +1 cc 0m 44s the patch passed
        +1 javac 0m 44s the patch passed
        -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 9 new + 252 unchanged - 12 fixed = 261 total (was 264)
        +1 mvnsite 0m 52s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 52s the patch passed
        +1 javadoc 0m 38s the patch passed
        -1 unit 106m 33s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        127m 48s



        Reason Tests
        Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839584/HDFS-11094.007.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 2e86695a51a5 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f6ffa11
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17607/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17607/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17607/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17607/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 28s trunk passed +1 compile 0m 56s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 0m 44s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 44s the patch passed +1 cc 0m 44s the patch passed +1 javac 0m 44s the patch passed -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 9 new + 252 unchanged - 12 fixed = 261 total (was 264) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 52s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 106m 33s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 127m 48s Reason Tests Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839584/HDFS-11094.007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 2e86695a51a5 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f6ffa11 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17607/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17607/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17607/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17607/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ebadger Eric Badger added a comment -

        Mingliang Liu, attaching a patch that includes unit tests both on the DN and NN side of the change. I mocked out most of it, so the tests should be pretty simple. But a review to make sure that I'm testing what I think I'm testing would be appreciated.

        Show
        ebadger Eric Badger added a comment - Mingliang Liu , attaching a patch that includes unit tests both on the DN and NN side of the change. I mocked out most of it, so the tests should be pretty simple. But a review to make sure that I'm testing what I think I'm testing would be appreciated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 7m 52s trunk passed
        +1 compile 0m 48s trunk passed
        +1 checkstyle 0m 30s trunk passed
        +1 mvnsite 1m 2s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 54s trunk passed
        +1 javadoc 0m 43s trunk passed
        +1 mvninstall 0m 57s the patch passed
        +1 compile 0m 52s the patch passed
        +1 cc 0m 52s the patch passed
        +1 javac 0m 52s the patch passed
        -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 10 new + 293 unchanged - 12 fixed = 303 total (was 305)
        +1 mvnsite 0m 57s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 48s the patch passed
        +1 javadoc 0m 37s the patch passed
        -1 unit 64m 13s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        84m 51s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestCrcCorruption
          hadoop.hdfs.server.namenode.TestDecommissioningStatus



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840122/HDFS-11094.008.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux ad91d461b11b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / beb70fe
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17639/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17639/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17639/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17639/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 52s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 57s the patch passed +1 compile 0m 52s the patch passed +1 cc 0m 52s the patch passed +1 javac 0m 52s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 10 new + 293 unchanged - 12 fixed = 303 total (was 305) +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 64m 13s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 84m 51s Reason Tests Failed junit tests hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.namenode.TestDecommissioningStatus Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840122/HDFS-11094.008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux ad91d461b11b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / beb70fe Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17639/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17639/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17639/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17639/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ebadger Eric Badger added a comment -

        Addressing checkstyle issues

        Show
        ebadger Eric Badger added a comment - Addressing checkstyle issues
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 30s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 7m 19s trunk passed
        +1 compile 0m 44s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 41s trunk passed
        +1 javadoc 0m 39s trunk passed
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 43s the patch passed
        +1 cc 0m 43s the patch passed
        +1 javac 0m 43s the patch passed
        -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 292 unchanged - 13 fixed = 294 total (was 305)
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 48s the patch passed
        +1 javadoc 0m 36s the patch passed
        +1 unit 61m 1s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        80m 16s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840254/HDFS-11094.009.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 9e6be85ea96c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3541ed8
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17648/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17648/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17648/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 30s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 19s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 43s the patch passed +1 cc 0m 43s the patch passed +1 javac 0m 43s the patch passed -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 292 unchanged - 13 fixed = 294 total (was 305) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 36s the patch passed +1 unit 61m 1s hadoop-hdfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 80m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840254/HDFS-11094.009.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 9e6be85ea96c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3541ed8 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17648/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17648/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17648/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment - - edited

        Hi Eric Badger, thanks for updating the patch. Sorry for returning late from holiday. The patch looks good to me overall. I have two thoughts for your consideration.

        1. I discussed with Arpit Agarwal offline and he suggested us use the same logic in updateActorStatesFromHeartbeat to update the active NN bpServiceToActive, which has dealt with several cases carefully. Moreover, if we are updating bpServiceToActive we should likely also update lastActiveClaimTxId. To achieve this, I think we can pass NNHAStatusHeartbeatProto instead of HAServiceStateProto in NamespaceInfoProto.
        2. For the unit test, can we set a very large heartbeat interval in configuration, and check the active NN is not null after cluster.waitForActive()? Mocked tests are useful as well and can be kept. Another idea is to drop heartbeat request against a spied HeartbeatManager.
        Show
        liuml07 Mingliang Liu added a comment - - edited Hi Eric Badger , thanks for updating the patch. Sorry for returning late from holiday. The patch looks good to me overall. I have two thoughts for your consideration. I discussed with Arpit Agarwal offline and he suggested us use the same logic in updateActorStatesFromHeartbeat to update the active NN bpServiceToActive , which has dealt with several cases carefully. Moreover, if we are updating bpServiceToActive we should likely also update lastActiveClaimTxId . To achieve this, I think we can pass NNHAStatusHeartbeatProto instead of HAServiceStateProto in NamespaceInfoProto . For the unit test, can we set a very large heartbeat interval in configuration, and check the active NN is not null after cluster.waitForActive() ? Mocked tests are useful as well and can be kept. Another idea is to drop heartbeat request against a spied HeartbeatManager.
        Hide
        ebadger Eric Badger added a comment -

        1. I discussed with Arpit Agarwal offline and he suggested us use the same logic in updateActorStatesFromHeartbeat to update the active NN bpServiceToActive, which has dealt with several cases carefully. Moreover, if we are updating bpServiceToActive we should likely also update lastActiveClaimTxId. To achieve this, I think we can pass NNHAStatusHeartbeatProto instead of HAServiceStateProto in NamespaceInfoProto.

        Mingliang Liu, I actually did it this way in the patch on purpose. The entire logic of updating bpServiceToActive will occur before any heartbeats start, since we are doing this during the handshake between the DN and the NN. If we send in an NNHAStatusHeartbeatProto instead of a HAServiceStateProto then we will have to deal with the lastActiveClaimTxId as you have mentioned. However, this would require more serious changes to the code, since we would have to either set and send along a TxId on the NN side (extra code change for what I see is negligible benefit) or we would need to arbitrarily create one on the DN side (would need to set it to be below the first heartbeat TxId, so it would have to be a negative number or would have to make extra changes).

        At this point, we want the DN to have an active before it starts trying to do anything with it (the whole point of this fix). If, for whatever reason, both NNs declare themselves as active, then it will choose the first one and ignore the second. If the wrong assertion is made, then it will talk to the standby and we will get a simple standby exception and then once the next heartbeat comes we will update the correct active. So worst case scenario we get a standby exception and retry, which is still loads better than the NPE that we were getting before. I think that since this is such a small window that it is unnecessary to make more changes with the TxId.

        Daryn Sharp may have more thoughts on this.

        2. For the unit test, can we set a very large heartbeat interval in configuration, and check the active NN is not null after cluster.waitForActive()? Mocked tests are useful as well and can be kept. Another idea is to drop heartbeat request against a spied HeartbeatManager.

        This should be fairly easy to do. I'll put up a patch shortly with this added test.

        Show
        ebadger Eric Badger added a comment - 1. I discussed with Arpit Agarwal offline and he suggested us use the same logic in updateActorStatesFromHeartbeat to update the active NN bpServiceToActive, which has dealt with several cases carefully. Moreover, if we are updating bpServiceToActive we should likely also update lastActiveClaimTxId. To achieve this, I think we can pass NNHAStatusHeartbeatProto instead of HAServiceStateProto in NamespaceInfoProto. Mingliang Liu , I actually did it this way in the patch on purpose. The entire logic of updating bpServiceToActive will occur before any heartbeats start, since we are doing this during the handshake between the DN and the NN. If we send in an NNHAStatusHeartbeatProto instead of a HAServiceStateProto then we will have to deal with the lastActiveClaimTxId as you have mentioned. However, this would require more serious changes to the code, since we would have to either set and send along a TxId on the NN side (extra code change for what I see is negligible benefit) or we would need to arbitrarily create one on the DN side (would need to set it to be below the first heartbeat TxId, so it would have to be a negative number or would have to make extra changes). At this point, we want the DN to have an active before it starts trying to do anything with it (the whole point of this fix). If, for whatever reason, both NNs declare themselves as active, then it will choose the first one and ignore the second. If the wrong assertion is made, then it will talk to the standby and we will get a simple standby exception and then once the next heartbeat comes we will update the correct active. So worst case scenario we get a standby exception and retry, which is still loads better than the NPE that we were getting before. I think that since this is such a small window that it is unnecessary to make more changes with the TxId. Daryn Sharp may have more thoughts on this. 2. For the unit test, can we set a very large heartbeat interval in configuration, and check the active NN is not null after cluster.waitForActive()? Mocked tests are useful as well and can be kept. Another idea is to drop heartbeat request against a spied HeartbeatManager. This should be fairly easy to do. I'll put up a patch shortly with this added test.
        Hide
        kihwal Kihwal Lee added a comment -

        For the unit test, can we set a very large heartbeat interval in configuration, and check the active NN is not null after cluster.waitForActive()?

        I think the existing tests are quite adequate. I understand that a full-blown mini cluster is sometimes needed to test the distributed file system. However, we should avoid adding such end-to-end tests if it is possible to have reasonable unit tests.

        Show
        kihwal Kihwal Lee added a comment - For the unit test, can we set a very large heartbeat interval in configuration, and check the active NN is not null after cluster.waitForActive()? I think the existing tests are quite adequate. I understand that a full-blown mini cluster is sometimes needed to test the distributed file system. However, we should avoid adding such end-to-end tests if it is possible to have reasonable unit tests.
        Hide
        ebadger Eric Badger added a comment -

        I think the existing tests are quite adequate. I understand that a full-blown mini cluster is sometimes needed to test the distributed file system. However, we should avoid adding such end-to-end tests if it is possible to have reasonable unit tests.

        Upon looking at this again, I agree with Kihwal Lee. I don't think that it is necessary for us to use a minicluster in this case. The current tests are adequate IMO since they test the methods that are directly used on either side of the version request. Additionally, the minicluster is expensive and creating a unit test with the minicluster would be difficult in this case since it requires a heartbeat to get out of its build() method (though difficulty is not my main objection).

        Show
        ebadger Eric Badger added a comment - I think the existing tests are quite adequate. I understand that a full-blown mini cluster is sometimes needed to test the distributed file system. However, we should avoid adding such end-to-end tests if it is possible to have reasonable unit tests. Upon looking at this again, I agree with Kihwal Lee . I don't think that it is necessary for us to use a minicluster in this case. The current tests are adequate IMO since they test the methods that are directly used on either side of the version request. Additionally, the minicluster is expensive and creating a unit test with the minicluster would be difficult in this case since it requires a heartbeat to get out of its build() method (though difficulty is not my main objection).
        Hide
        liuml07 Mingliang Liu added a comment -

        For the protocol changes, end-to-end tests are very helpful. Starting a mini dfs cluster is not very expensive; I can usually finish start and shutdown an empty mini cluster in 3~5 seconds on my dev machine.

        The first heartbeat will bypass the large interval; so 1) Choosing HAServiceStateProto instead of HAServiceStateProto makes sense as lastActiveClaimTxId will be updated in a timely manner, and we can save the complexity of updating it in this patch; 2) Unfortunately, current methods (e.g. set large config DFS_HEARTBEAT_INTERVAL_KEY, or DataNode#setHeartbeatsDisabledForTests()) are not working without change for testing this patch. I can accept that existing tests in patch are somehow adequate. So this will not block the progress of this patch.

        Thanks,

        Show
        liuml07 Mingliang Liu added a comment - For the protocol changes, end-to-end tests are very helpful. Starting a mini dfs cluster is not very expensive; I can usually finish start and shutdown an empty mini cluster in 3~5 seconds on my dev machine. The first heartbeat will bypass the large interval; so 1) Choosing HAServiceStateProto instead of HAServiceStateProto makes sense as lastActiveClaimTxId will be updated in a timely manner, and we can save the complexity of updating it in this patch; 2) Unfortunately, current methods (e.g. set large config DFS_HEARTBEAT_INTERVAL_KEY , or DataNode#setHeartbeatsDisabledForTests() ) are not working without change for testing this patch. I can accept that existing tests in patch are somehow adequate. So this will not block the progress of this patch. Thanks,
        Hide
        daryn Daryn Sharp added a comment -

        +1 Assuming no other blocking objections. I've been involved with the patch offline.

        Show
        daryn Daryn Sharp added a comment - +1 Assuming no other blocking objections. I've been involved with the patch offline.
        Hide
        liuml07 Mingliang Liu added a comment -

        I've just committed to trunk branch. I see conflicts when cherry-picking to branch-2. Eric Badger, can you provide a patch for branch-2 if this should go to branch-2 as well? Thanks,

        Show
        liuml07 Mingliang Liu added a comment - I've just committed to trunk branch. I see conflicts when cherry-picking to branch-2 . Eric Badger , can you provide a patch for branch-2 if this should go to branch-2 as well? Thanks,
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10939 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10939/)
        HDFS-11094. Send back HAState along with NamespaceInfo during a (liuml07: rev 8c4680852b20ad0e65e77dd123c9ba5bb6f2fa39)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/HdfsServer.proto
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10939 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10939/ ) HDFS-11094 . Send back HAState along with NamespaceInfo during a (liuml07: rev 8c4680852b20ad0e65e77dd123c9ba5bb6f2fa39) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/HdfsServer.proto (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        ebadger Eric Badger added a comment -

        Mingliang Liu, attaching a branch-2/branch-2.8 patch. Just had to change around the type definitions of some things. Also moved NNHAStatusHeartbeatProto from DatanodeProtocol.proto to HdfsServer.proto (which is imported by DatanodeProtocol.proto) so that it could be used by NamespaceInfoProto.

        Show
        ebadger Eric Badger added a comment - Mingliang Liu , attaching a branch-2/branch-2.8 patch. Just had to change around the type definitions of some things. Also moved NNHAStatusHeartbeatProto from DatanodeProtocol.proto to HdfsServer.proto (which is imported by DatanodeProtocol.proto) so that it could be used by NamespaceInfoProto .
        Hide
        ebadger Eric Badger added a comment -

        Mingliang Liu, actually hold off on committing that branch-2/branch-2.8 patch. Can you instead revert the trunk commit? HDFS-11207 looks like it will probably revert HDFS-5079. However, we will need to revert this jira first to avoid breaking the build. After HDFS-5079 gets reverted, we should be able to use 1 patch (the branch-2/branch-2.8 patch) to commit all the way through from trunk.

        Show
        ebadger Eric Badger added a comment - Mingliang Liu , actually hold off on committing that branch-2/branch-2.8 patch. Can you instead revert the trunk commit? HDFS-11207 looks like it will probably revert HDFS-5079 . However, we will need to revert this jira first to avoid breaking the build. After HDFS-5079 gets reverted, we should be able to use 1 patch (the branch-2/branch-2.8 patch) to commit all the way through from trunk.
        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks for bringing this up. If we agree to revert HDFS-5079 in HDFS-11207, we can firstly revert this to avoid breaking the build in trunk. We need more input who evolved in HDFS-5079 before reverting it. For branch-2 I won't commit the current patch.

        Show
        liuml07 Mingliang Liu added a comment - Thanks for bringing this up. If we agree to revert HDFS-5079 in HDFS-11207 , we can firstly revert this to avoid breaking the build in trunk . We need more input who evolved in HDFS-5079 before reverting it. For branch-2 I won't commit the current patch.
        Hide
        ebadger Eric Badger added a comment -

        Ok, yes, that sounds good. Thanks!

        Show
        ebadger Eric Badger added a comment - Ok, yes, that sounds good. Thanks!
        Hide
        kihwal Kihwal Lee added a comment -

        I've reverted this, as HDFS-5079 was reverted. Please rework the patch to reflect that.

        Show
        kihwal Kihwal Lee added a comment - I've reverted this, as HDFS-5079 was reverted. Please rework the patch to reflect that.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10973 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10973/)
        Revert "HDFS-11094. Send back HAState along with NamespaceInfo during a (kihwal: rev 9f8344db502b833efe0f2c554b67098e77063d20)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/HdfsServer.proto
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10973 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10973/ ) Revert " HDFS-11094 . Send back HAState along with NamespaceInfo during a (kihwal: rev 9f8344db502b833efe0f2c554b67098e77063d20) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/HdfsServer.proto (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
        Hide
        ebadger Eric Badger added a comment -

        Attaching new trunk patch after the revert

        Show
        ebadger Eric Badger added a comment - Attaching new trunk patch after the revert
        Hide
        ebadger Eric Badger added a comment -

        Attaching associated branch-2/branch-2.8 patch since it won't cherry-pick cleanly from trunk.

        Show
        ebadger Eric Badger added a comment - Attaching associated branch-2/branch-2.8 patch since it won't cherry-pick cleanly from trunk.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 5s HDFS-11094 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue HDFS-11094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842440/HDFS-11094.010-b2.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17801/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HDFS-11094 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-11094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842440/HDFS-11094.010-b2.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17801/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ebadger Eric Badger added a comment -

        Attaching trunk patch again so it runs against jenkins instead of the branch-2 patch

        Show
        ebadger Eric Badger added a comment - Attaching trunk patch again so it runs against jenkins instead of the branch-2 patch
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 46s trunk passed
        +1 compile 0m 45s trunk passed
        +1 checkstyle 0m 30s trunk passed
        +1 mvnsite 0m 50s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 40s trunk passed
        +1 javadoc 0m 39s trunk passed
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 41s the patch passed
        +1 cc 0m 41s the patch passed
        +1 javac 0m 41s the patch passed
        -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 292 unchanged - 13 fixed = 294 total (was 305)
        +1 mvnsite 0m 47s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 45s the patch passed
        +1 javadoc 0m 37s the patch passed
        -1 unit 61m 42s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        80m 3s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestBPOfferService
          hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842442/HDFS-11094.010.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux df02408c4daf 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 13d8e55
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17802/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17802/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17802/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17802/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 46s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 41s the patch passed +1 cc 0m 41s the patch passed +1 javac 0m 41s the patch passed -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 292 unchanged - 13 fixed = 294 total (was 305) +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 61m 42s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 80m 3s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestBPOfferService   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842442/HDFS-11094.010.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux df02408c4daf 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 13d8e55 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17802/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17802/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17802/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17802/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ebadger Eric Badger added a comment -

        The TestBPOfferService failure is definitely relevant here. Not sure about the other one. Let me take a look

        Show
        ebadger Eric Badger added a comment - The TestBPOfferService failure is definitely relevant here. Not sure about the other one. Let me take a look
        Hide
        ebadger Eric Badger added a comment -

        The test was racy. Heartbeats were setting the active NN to null after it was getting set by the test. Fixed the test by turning off heartbeats. The other unit test is failing elsewhere and not related to this patch.

        Show
        ebadger Eric Badger added a comment - The test was racy. Heartbeats were setting the active NN to null after it was getting set by the test. Fixed the test by turning off heartbeats. The other unit test is failing elsewhere and not related to this patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 7m 48s trunk passed
        +1 compile 0m 51s trunk passed
        +1 checkstyle 0m 34s trunk passed
        +1 mvnsite 1m 0s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 1m 51s trunk passed
        +1 javadoc 0m 41s trunk passed
        +1 mvninstall 0m 48s the patch passed
        +1 compile 0m 51s the patch passed
        +1 cc 0m 51s the patch passed
        +1 javac 0m 51s the patch passed
        -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 292 unchanged - 13 fixed = 294 total (was 305)
        +1 mvnsite 0m 54s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 53s the patch passed
        +1 javadoc 0m 38s the patch passed
        -1 unit 94m 5s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        114m 49s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.TestReconstructStripedBlocks



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842581/HDFS-11094.011.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 5726f6a78517 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 80b8023
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17809/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17809/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17809/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17809/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 48s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 51s trunk passed +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 51s the patch passed +1 cc 0m 51s the patch passed +1 javac 0m 51s the patch passed -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 292 unchanged - 13 fixed = 294 total (was 305) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 94m 5s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 114m 49s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestReconstructStripedBlocks Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842581/HDFS-11094.011.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 5726f6a78517 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 80b8023 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17809/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17809/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17809/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17809/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ebadger Eric Badger added a comment -

        Test failure looks like it's unrelated and doesn't fail for me locally

        Show
        ebadger Eric Badger added a comment - Test failure looks like it's unrelated and doesn't fail for me locally
        Hide
        ebadger Eric Badger added a comment -

        Mingliang Liu, can you take a look at the latest patch?

        Show
        ebadger Eric Badger added a comment - Mingliang Liu , can you take a look at the latest patch?
        Hide
        liuml07 Mingliang Liu added a comment -

        Yes I will review this week if no reviews from others. Thanks,

        Show
        liuml07 Mingliang Liu added a comment - Yes I will review this week if no reviews from others. Thanks,
        Hide
        liuml07 Mingliang Liu added a comment -

        +1

        Failing test is not related. Do we need separate patch for branch-2?

        Daryn Sharp and Kihwal Lee if you don't have further comment, I'll commit the v12 patch later this week. Thanks,

        Show
        liuml07 Mingliang Liu added a comment - +1 Failing test is not related. Do we need separate patch for branch-2 ? Daryn Sharp and Kihwal Lee if you don't have further comment, I'll commit the v12 patch later this week. Thanks,
        Hide
        kihwal Kihwal Lee added a comment -

        I am fine with the latest patch.

        Show
        kihwal Kihwal Lee added a comment - I am fine with the latest patch.
        Hide
        ebadger Eric Badger added a comment -

        Uploading branch-2 patch.

        Show
        ebadger Eric Badger added a comment - Uploading branch-2 patch.
        Hide
        liuml07 Mingliang Liu added a comment -

        +1 pending on Jenkins.

        Show
        liuml07 Mingliang Liu added a comment - +1 pending on Jenkins.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 13m 26s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 36s branch-2 passed
        +1 compile 0m 38s branch-2 passed with JDK v1.8.0_111
        +1 compile 0m 42s branch-2 passed with JDK v1.7.0_121
        +1 checkstyle 0m 31s branch-2 passed
        +1 mvnsite 0m 51s branch-2 passed
        +1 mvneclipse 0m 16s branch-2 passed
        +1 findbugs 1m 55s branch-2 passed
        +1 javadoc 0m 53s branch-2 passed with JDK v1.8.0_111
        +1 javadoc 1m 36s branch-2 passed with JDK v1.7.0_121
        +1 mvninstall 0m 44s the patch passed
        +1 compile 0m 36s the patch passed with JDK v1.8.0_111
        +1 cc 0m 36s the patch passed
        +1 javac 0m 36s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.7.0_121
        +1 cc 0m 39s the patch passed
        +1 javac 0m 39s the patch passed
        -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 300 unchanged - 13 fixed = 302 total (was 313)
        +1 mvnsite 0m 49s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 7s the patch passed
        +1 javadoc 0m 50s the patch passed with JDK v1.8.0_111
        +1 javadoc 1m 33s the patch passed with JDK v1.7.0_121
        -1 unit 49m 50s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        140m 24s



        Reason Tests
        JDK v1.8.0_111 Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache
        JDK v1.7.0_121 Failed junit tests hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b59b8b7
        JIRA Issue HDFS-11094
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843465/HDFS-11094-branch-2.011.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 386b3b1ac5b7 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 8673892
        Default Java 1.7.0_121
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17868/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17868/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
        JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17868/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17868/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 13m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 36s branch-2 passed +1 compile 0m 38s branch-2 passed with JDK v1.8.0_111 +1 compile 0m 42s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 31s branch-2 passed +1 mvnsite 0m 51s branch-2 passed +1 mvneclipse 0m 16s branch-2 passed +1 findbugs 1m 55s branch-2 passed +1 javadoc 0m 53s branch-2 passed with JDK v1.8.0_111 +1 javadoc 1m 36s branch-2 passed with JDK v1.7.0_121 +1 mvninstall 0m 44s the patch passed +1 compile 0m 36s the patch passed with JDK v1.8.0_111 +1 cc 0m 36s the patch passed +1 javac 0m 36s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_121 +1 cc 0m 39s the patch passed +1 javac 0m 39s the patch passed -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 300 unchanged - 13 fixed = 302 total (was 313) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 7s the patch passed +1 javadoc 0m 50s the patch passed with JDK v1.8.0_111 +1 javadoc 1m 33s the patch passed with JDK v1.7.0_121 -1 unit 49m 50s hadoop-hdfs in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 140m 24s Reason Tests JDK v1.8.0_111 Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache JDK v1.7.0_121 Failed junit tests hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-11094 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843465/HDFS-11094-branch-2.011.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 386b3b1ac5b7 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 8673892 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17868/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17868/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17868/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17868/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment -

        Test failures are not related.

        Committed to branch-2 and trunk branches. Resolving this (again). Thanks all for the contribution.

        Show
        liuml07 Mingliang Liu added a comment - Test failures are not related. Committed to branch-2 and trunk branches. Resolving this (again). Thanks all for the contribution.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11004 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11004/)
        HDFS-11094. Send back HAState along with NamespaceInfo during a (liuml07: rev ac0d402c6a3af784aa8bdceecf3293c30760346f)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/HdfsServer.proto
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11004 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11004/ ) HDFS-11094 . Send back HAState along with NamespaceInfo during a (liuml07: rev ac0d402c6a3af784aa8bdceecf3293c30760346f) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/HdfsServer.proto (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
        Hide
        ebadger Eric Badger added a comment -

        Thanks, Mingliang Liu!

        Show
        ebadger Eric Badger added a comment - Thanks, Mingliang Liu !
        Hide
        ebadger Eric Badger added a comment -

        Mingliang Liu, can we cherry-pick this to 2.8? I'm seeing test failures from TestLargeBlockReport.testBlockReportSucceedsWithLargerLengthLimit due to a race condition in getActiveNN() that this will fix.

        Show
        ebadger Eric Badger added a comment - Mingliang Liu , can we cherry-pick this to 2.8? I'm seeing test failures from TestLargeBlockReport.testBlockReportSucceedsWithLargerLengthLimit due to a race condition in getActiveNN() that this will fix.
        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks, Eric Badger. I've backported to branch-2.8 branch.

        Show
        liuml07 Mingliang Liu added a comment - Thanks, Eric Badger . I've backported to branch-2.8 branch.

          People

          • Assignee:
            ebadger Eric Badger
            Reporter:
            ebadger Eric Badger
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development