Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6533

TestBPOfferService#testBasicFunctionalitytest fails intermittently

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Per https://builds.apache.org/job/Hadoop-Hdfs-trunk/1774/testReport, the following test failed. However, local rerun is successful.

      org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testBasicFunctionality
      
      Error Message
      
      Wanted but not invoked:
      datanodeProtocolClientSideTranslatorPB.registerDatanode(
          <any>
      );
      -> at org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testBasicFunctionality(TestBPOfferService.java:175)
      Actually, there were zero interactions with this mock.
      Stacktrace
      
      org.mockito.exceptions.verification.WantedButNotInvoked: 
      Wanted but not invoked:
      datanodeProtocolClientSideTranslatorPB.registerDatanode(
          <any>
      );
      -> at org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testBasicFunctionality(TestBPOfferService.java:175)
      Actually, there were zero interactions with this mock.
      
      	at org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testBasicFunctionality(TestBPOfferService.java:175)
      Standard Output
      
      2014-06-14 12:42:08,723 INFO  datanode.DataNode (SimulatedFSDataset.java:registerMBean(968)) - Registered FSDatasetState MBean
      2014-06-14 12:42:08,730 INFO  datanode.DataNode (BPServiceActor.java:run(805)) - Block pool <registering> (Datanode Uuid unassigned) service to 0.0.0.0/0.0.0.0:0 starting to offer service
      2014-06-14 12:42:08,730 DEBUG datanode.DataNode (BPServiceActor.java:retrieveNamespaceInfo(170)) - Block pool <registering> (Datanode Uuid unassigned) service to 0.0.0.0/0.0.0.0:0 received versionRequest response: lv=-57;cid=fake cluster;nsid=1;c=0;bpid=fake bpid
      2014-06-14 12:42:08,731 INFO  datanode.DataNode (BPServiceActor.java:register(765)) - Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:0 beginning handshake with NN
      2014-06-14 12:42:08,731 INFO  datanode.DataNode (BPServiceActor.java:register(778)) - Block pool Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:0 successfully registered with NN
      2014-06-14 12:42:08,732 INFO  datanode.DataNode (BPServiceActor.java:offerService(637)) - For namenode 0.0.0.0/0.0.0.0:0 using DELETEREPORT_INTERVAL of 300000 msec  BLOCKREPORT_INTERVAL of 21600000msec CACHEREPORT_INTERVAL of 10000msec Initial delay: 0msec; heartBeatInterval=3000
      2014-06-14 12:42:08,732 DEBUG datanode.DataNode (BPServiceActor.java:sendHeartBeat(562)) - Sending heartbeat with 1 storage reports from service actor: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:0
      2014-06-14 12:42:08,734 INFO  datanode.DataNode (BPServiceActor.java:blockReport(498)) - Sent 1 blockreports 0 blocks total. Took 1 msec to generate and 0 msecs for RPC and NN processing.  Got back commands none
      2014-06-14 12:42:08,738 INFO  datanode.DataNode (BPServiceActor.java:run(805)) - Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:1 starting to offer service
      2014-06-14 12:42:08,739 DEBUG datanode.DataNode (BPServiceActor.java:retrieveNamespaceInfo(170)) - Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:1 received versionRequest response: lv=-57;cid=fake cluster;nsid=1;c=0;bpid=fake bpid
      2014-06-14 12:42:08,739 INFO  datanode.DataNode (BPServiceActor.java:register(765)) - Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:1 beginning handshake with NN
      2014-06-14 12:42:08,740 INFO  datanode.DataNode (BPServiceActor.java:register(778)) - Block pool Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:1 successfully registered with NN
      
      1. HDFS-6533.004.patch
        4 kB
        Wei-Chiu Chuang
      2. HDFS-6533.003.patch
        5 kB
        Wei-Chiu Chuang
      3. HDFS-6533.002.patch
        4 kB
        Wei-Chiu Chuang
      4. HDFS-6533.001.patch
        5 kB
        Wei-Chiu Chuang

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #656 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/656/)
        HDFS-6533. TestBPOfferService#testBasicFunctionalitytest fails (arp: rev 58f6f54eeac779428ac995d196b60ffb90563f97)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #656 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/656/ ) HDFS-6533 . TestBPOfferService#testBasicFunctionalitytest fails (arp: rev 58f6f54eeac779428ac995d196b60ffb90563f97) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Done. Thanks Kihwal Lee.

        Show
        arpitagarwal Arpit Agarwal added a comment - Done. Thanks Kihwal Lee .
        Hide
        kihwal Kihwal Lee added a comment -

        Arpit Agarwal Vinod has created branch-2.8. So branch-2 is 2.9 now.

        Show
        kihwal Kihwal Lee added a comment - Arpit Agarwal Vinod has created branch-2.8. So branch-2 is 2.9 now.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #8907 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8907/)
        HDFS-6533. TestBPOfferService#testBasicFunctionalitytest fails (arp: rev 58f6f54eeac779428ac995d196b60ffb90563f97)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #8907 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8907/ ) HDFS-6533 . TestBPOfferService#testBasicFunctionalitytest fails (arp: rev 58f6f54eeac779428ac995d196b60ffb90563f97) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks Arpit Agarwal for the code review and commit. Learned a lot from you.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks Arpit Agarwal for the code review and commit. Learned a lot from you.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Committed to trunk and branch-2. Thank you for the contribution Wei-Chiu Chuang.

        Show
        arpitagarwal Arpit Agarwal added a comment - Committed to trunk and branch-2. Thank you for the contribution Wei-Chiu Chuang .
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Test failures are unrelated.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Test failures are unrelated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 26s trunk passed
        +1 compile 0m 52s trunk passed with JDK v1.8.0_66
        +1 compile 0m 49s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 58s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 2m 10s trunk passed
        +1 javadoc 1m 16s trunk passed with JDK v1.8.0_66
        +1 javadoc 2m 1s trunk passed with JDK v1.7.0_85
        +1 mvninstall 0m 53s the patch passed
        +1 compile 0m 50s the patch passed with JDK v1.8.0_66
        +1 javac 0m 50s the patch passed
        +1 compile 0m 49s the patch passed with JDK v1.7.0_85
        +1 javac 0m 49s the patch passed
        +1 checkstyle 0m 18s the patch passed
        +1 mvnsite 1m 4s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 34s the patch passed
        +1 javadoc 1m 19s the patch passed with JDK v1.8.0_66
        +1 javadoc 2m 12s the patch passed with JDK v1.7.0_85
        -1 unit 64m 30s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 57m 38s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
        -1 asflicense 0m 20s Patch generated 56 ASF License warnings.
        152m 52s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          hadoop.hdfs.TestDistributedFileSystem
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.server.datanode.TestBlockReplacement
        JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.datanode.TestBlockReplacement



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775078/HDFS-6533.004.patch
        JIRA Issue HDFS-6533
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3da6097e470e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 485c346
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13714/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13714/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13714/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13714/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13714/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13714/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13714/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 26s trunk passed +1 compile 0m 52s trunk passed with JDK v1.8.0_66 +1 compile 0m 49s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 10s trunk passed +1 javadoc 1m 16s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 1s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 53s the patch passed +1 compile 0m 50s the patch passed with JDK v1.8.0_66 +1 javac 0m 50s the patch passed +1 compile 0m 49s the patch passed with JDK v1.7.0_85 +1 javac 0m 49s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 34s the patch passed +1 javadoc 1m 19s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 12s the patch passed with JDK v1.7.0_85 -1 unit 64m 30s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 57m 38s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 20s Patch generated 56 ASF License warnings. 152m 52s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.datanode.TestBlockReplacement JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.datanode.TestBlockReplacement Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775078/HDFS-6533.004.patch JIRA Issue HDFS-6533 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3da6097e470e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 485c346 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13714/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13714/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13714/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13714/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13714/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13714/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13714/console This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        +1 for the v4 patch, pending Jenkins.

        Show
        arpitagarwal Arpit Agarwal added a comment - +1 for the v4 patch, pending Jenkins.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Rev04: code refactoring per Arpit Agarwal. I am running locally and not observing test failures.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Rev04: code refactoring per Arpit Agarwal . I am running locally and not observing test failures.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks for pointing that out. Agree that minimizing change in DN is the right direction. I did not notice there is already a BPOfferService#getBPServiceActors(). I'll update soon.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks for pointing that out. Agree that minimizing change in DN is the right direction. I did not notice there is already a BPOfferService#getBPServiceActors(). I'll update soon.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Hi Wei-Chiu Chuang, thanks for the updated patch. I think we can just move BPOfferService#countRegisteredActors to the test case and get the list of actors via BPOfferService#getBPServiceActors. You may just have to add a getter for BPServiceActor#bpRegistration.

        The reason for minimizing new logic in the DN classes is that outside of the test case, countRegisteredActors and isRegistered may be misleading. e.g. if the NN has asked the DN to re-register we don't null the existing registration. (see BPServiceActor#processCommandFromActor and #reRegister).

        Show
        arpitagarwal Arpit Agarwal added a comment - Hi Wei-Chiu Chuang , thanks for the updated patch. I think we can just move BPOfferService#countRegisteredActors to the test case and get the list of actors via BPOfferService#getBPServiceActors. You may just have to add a getter for BPServiceActor#bpRegistration. The reason for minimizing new logic in the DN classes is that outside of the test case, countRegisteredActors and isRegistered may be misleading. e.g. if the NN has asked the DN to re-register we don't null the existing registration. (see BPServiceActor#processCommandFromActor and #reRegister).
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Test failures are unrelated. License warnings are due to temp files and can be safely ignored.

        I also ran the patched test for 1000 times on my local machines without failures. Previously, it fails 1 time in about 50 runs.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Test failures are unrelated. License warnings are due to temp files and can be safely ignored. I also ran the patched test for 1000 times on my local machines without failures. Previously, it fails 1 time in about 50 runs.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 34s trunk passed
        +1 compile 0m 48s trunk passed with JDK v1.8.0_66
        +1 compile 0m 45s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 56s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 2m 11s trunk passed
        +1 javadoc 1m 14s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 56s trunk passed with JDK v1.7.0_85
        +1 mvninstall 0m 54s the patch passed
        +1 compile 0m 46s the patch passed with JDK v1.8.0_66
        +1 javac 0m 46s the patch passed
        +1 compile 0m 45s the patch passed with JDK v1.7.0_85
        +1 javac 0m 45s the patch passed
        +1 checkstyle 0m 17s the patch passed
        +1 mvnsite 1m 3s the patch passed
        +1 mvneclipse 0m 18s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 37s the patch passed
        +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 52s the patch passed with JDK v1.7.0_85
        -1 unit 57m 18s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 54m 1s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
        -1 asflicense 0m 20s Patch generated 58 ASF License warnings.
        141m 8s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestReadStripedFileWithDecoding
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          hadoop.hdfs.server.datanode.TestBlockScanner
        JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774940/HDFS-6533.003.patch
        JIRA Issue HDFS-6533
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux bb6b2a64635c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1c05393
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13700/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13700/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13700/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 34s trunk passed +1 compile 0m 48s trunk passed with JDK v1.8.0_66 +1 compile 0m 45s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 11s trunk passed +1 javadoc 1m 14s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 56s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 54s the patch passed +1 compile 0m 46s the patch passed with JDK v1.8.0_66 +1 javac 0m 46s the patch passed +1 compile 0m 45s the patch passed with JDK v1.7.0_85 +1 javac 0m 45s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 37s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 52s the patch passed with JDK v1.7.0_85 -1 unit 57m 18s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 54m 1s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 20s Patch generated 58 ASF License warnings. 141m 8s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.datanode.TestBlockScanner JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774940/HDFS-6533.003.patch JIRA Issue HDFS-6533 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bb6b2a64635c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1c05393 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13700/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13700/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13700/console This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks Arpit Agarwal for the comments. Yes that looks to be a better idea.

        I am attaching rev03 that follows your suggestion.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks Arpit Agarwal for the comments. Yes that looks to be a better idea. I am attaching rev03 that follows your suggestion.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 10m 24s trunk passed
        +1 compile 1m 10s trunk passed with JDK v1.8.0_66
        +1 compile 0m 59s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 1m 12s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 2m 35s trunk passed
        +1 javadoc 1m 39s trunk passed with JDK v1.8.0_66
        +1 javadoc 2m 26s trunk passed with JDK v1.7.0_85
        +1 mvninstall 1m 6s the patch passed
        +1 compile 1m 11s the patch passed with JDK v1.8.0_66
        +1 javac 1m 11s the patch passed
        +1 compile 0m 59s the patch passed with JDK v1.7.0_85
        +1 javac 0m 59s the patch passed
        -1 checkstyle 0m 23s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 23, now 24).
        +1 mvnsite 1m 16s the patch passed
        +1 mvneclipse 0m 19s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        -1 findbugs 2m 55s hadoop-hdfs-project/hadoop-hdfs introduced 1 new FindBugs issues.
        +1 javadoc 1m 43s the patch passed with JDK v1.8.0_66
        +1 javadoc 2m 29s the patch passed with JDK v1.7.0_85
        -1 unit 87m 1s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 74m 56s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
        -1 asflicense 0m 25s Patch generated 58 ASF License warnings.
        199m 40s



        Reason Tests
        FindBugs module:hadoop-hdfs-project/hadoop-hdfs
          Increment of volatile field org.apache.hadoop.hdfs.server.datanode.BPOfferService.registeredActors in org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPServiceActor, DatanodeRegistration) At BPOfferService.java:in org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPServiceActor, DatanodeRegistration) At BPOfferService.java:[line 371]
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          hadoop.fs.TestUnbuffer
          hadoop.hdfs.TestHFlush
          hadoop.hdfs.server.datanode.TestBlockReplacement
          hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          hadoop.hdfs.TestDFSClientRetries
          hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
        JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestEncryptionZones



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774904/HDFS-6533.002.patch
        JIRA Issue HDFS-6533
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3e6a1f31ef34 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9b8e50b
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13690/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13690/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13690/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13690/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13690/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13690/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13690/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13690/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 75MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13690/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 10m 24s trunk passed +1 compile 1m 10s trunk passed with JDK v1.8.0_66 +1 compile 0m 59s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 24s trunk passed +1 mvnsite 1m 12s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 2m 35s trunk passed +1 javadoc 1m 39s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 26s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 6s the patch passed +1 compile 1m 11s the patch passed with JDK v1.8.0_66 +1 javac 1m 11s the patch passed +1 compile 0m 59s the patch passed with JDK v1.7.0_85 +1 javac 0m 59s the patch passed -1 checkstyle 0m 23s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 23, now 24). +1 mvnsite 1m 16s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 2m 55s hadoop-hdfs-project/hadoop-hdfs introduced 1 new FindBugs issues. +1 javadoc 1m 43s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 29s the patch passed with JDK v1.7.0_85 -1 unit 87m 1s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 74m 56s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 25s Patch generated 58 ASF License warnings. 199m 40s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Increment of volatile field org.apache.hadoop.hdfs.server.datanode.BPOfferService.registeredActors in org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPServiceActor, DatanodeRegistration) At BPOfferService.java:in org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPServiceActor, DatanodeRegistration) At BPOfferService.java: [line 371] JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.fs.TestUnbuffer   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestEncryptionZones Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774904/HDFS-6533.002.patch JIRA Issue HDFS-6533 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3e6a1f31ef34 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9b8e50b findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13690/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13690/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13690/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13690/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13690/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13690/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13690/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13690/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13690/console This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Thanks for taking this up Wei-Chiu Chuang.

        Instead of adding a new fields to BpOfferService, can we rewrite the check in waitForBothActors to verify both actors have a valid bpRegistration? It is initialized on successful registration.

        Show
        arpitagarwal Arpit Agarwal added a comment - Thanks for taking this up Wei-Chiu Chuang . Instead of adding a new fields to BpOfferService, can we rewrite the check in waitForBothActors to verify both actors have a valid bpRegistration ? It is initialized on successful registration.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Rev02.

        I ran rev01 code locally for a few thousand times, and it looks like the test did not fail at all. However, another test failed because it explicitly tested the scenario where one name node failed and wanted to wait for just one actor. So I created a separate utility method exclusively for testBasicFunctionality that waits for both actor registrations, and waitForInitialization() only waits for one actor registration.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Rev02. I ran rev01 code locally for a few thousand times, and it looks like the test did not fail at all. However, another test failed because it explicitly tested the scenario where one name node failed and wanted to wait for just one actor. So I created a separate utility method exclusively for testBasicFunctionality that waits for both actor registrations, and waitForInitialization() only waits for one actor registration.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Rev01: proof of concept.

        There are two issues here:
        (1) Upon exiting, the test interrupts BPServiceActor threads, but does not wait for them to finish. This is error-prone, and makes the output of subsequent tests mix with previous ones.

        (2) The direct cause of test failure is that the test does not properly wait for actors register with both name nodes. The method BPOfferService.isInitialized() returns true if any of the actors register with the corresponding name nodes. I modified the wait-for condition to explicitly wait for the registrations with both name nodes.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Rev01: proof of concept. There are two issues here: (1) Upon exiting, the test interrupts BPServiceActor threads, but does not wait for them to finish. This is error-prone, and makes the output of subsequent tests mix with previous ones. (2) The direct cause of test failure is that the test does not properly wait for actors register with both name nodes. The method BPOfferService.isInitialized() returns true if any of the actors register with the corresponding name nodes. I modified the wait-for condition to explicitly wait for the registrations with both name nodes.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks Yongjun Zhang I have observed this issue locally and would like to begin working on this.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks Yongjun Zhang I have observed this issue locally and would like to begin working on this.

          People

          • Assignee:
            jojochuang Wei-Chiu Chuang
            Reporter:
            yzhangal Yongjun Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development