Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2700

TestDataNodeMultipleRegistrations is failing in trunk

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestDataNodeMultipleRegistrations is failing from last couple of builds
      https://builds.apache.org/job/PreCommit-HDFS-Build/lastCompletedBuild/testReport/

      1. HDFS-2700.patch
        1 kB
        Uma Maheswara Rao G

        Activity

        Hide
        Uma Maheswara Rao G added a comment -

        more info:
        java.io.IOException: Failed on local exception: java.io.EOFException; Host Details : local host is: "asf001.sp2.ygridcore.net/67.195.138.31"; destination host is: ""localhost":9929;
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:655)
        at org.apache.hadoop.ipc.Client.call(Client.java:1140)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:169)
        at $Proxy14.getDatanodeReport(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:127)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:81)
        at $Proxy14.getDatanodeReport(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDatanodeReport(ClientNamenodeProtocolTranslatorPB.java:555)
        at org.apache.hadoop.hdfs.DFSClient.datanodeReport(DFSClient.java:1443)
        at org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:1486)
        at org.apache.hadoop.hdfs.MiniDFSCluster.addNameNode(MiniDFSCluster.java:1904)
        at org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations.testMiniDFSClusterWithMultipleNN(TestDataNodeMultipleRegistrations.java:237)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        Show
        Uma Maheswara Rao G added a comment - more info: java.io.IOException: Failed on local exception: java.io.EOFException; Host Details : local host is: "asf001.sp2.ygridcore.net/67.195.138.31"; destination host is: ""localhost":9929; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:655) at org.apache.hadoop.ipc.Client.call(Client.java:1140) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:169) at $Proxy14.getDatanodeReport(Unknown Source) at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:127) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:81) at $Proxy14.getDatanodeReport(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDatanodeReport(ClientNamenodeProtocolTranslatorPB.java:555) at org.apache.hadoop.hdfs.DFSClient.datanodeReport(DFSClient.java:1443) at org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:1486) at org.apache.hadoop.hdfs.MiniDFSCluster.addNameNode(MiniDFSCluster.java:1904) at org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations.testMiniDFSClusterWithMultipleNN(TestDataNodeMultipleRegistrations.java:237) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        Hide
        Uma Maheswara Rao G added a comment -

        Some more information:

        Looks nor able clean the some of the proxy instances

        BP-239265342-67.195.138.31-1324137379885 (storage id DS-47228547-67.195.138.31-49285-1324137385405) registered with localhost/127.0.0.1:9930
        2011-12-17 15:56:26,248 ERROR ipc.RPC (RPC.java:stopProxy(559)) - Tried to call RPC.stopProxy on an object that is not a proxy.
        java.lang.IllegalArgumentException: not a proxy instance
        at java.lang.reflect.Proxy.getInvocationHandler(Proxy.java:637)
        at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.cleanUp(BPOfferService.java:450)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.run(BPOfferService.java:639)
        at java.lang.Thread.run(Thread.java:662)
        2011-12-17 15:56:26,248 ERROR ipc.RPC (RPC.java:stopProxy(559)) - Tried to call RPC.stopProxy on an object that is not a proxy.
        java.lang.IllegalArgumentException: not a proxy instance
        at java.lang.reflect.Proxy.getInvocationHandler(Proxy.java:637)
        at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.cleanUp(BPOfferService.java:450)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.run(BPOfferService.java:639)
        at java.lang.Thread.run(Thread.java:662)

        Show
        Uma Maheswara Rao G added a comment - Some more information: Looks nor able clean the some of the proxy instances BP-239265342-67.195.138.31-1324137379885 (storage id DS-47228547-67.195.138.31-49285-1324137385405) registered with localhost/127.0.0.1:9930 2011-12-17 15:56:26,248 ERROR ipc.RPC (RPC.java:stopProxy(559)) - Tried to call RPC.stopProxy on an object that is not a proxy. java.lang.IllegalArgumentException: not a proxy instance at java.lang.reflect.Proxy.getInvocationHandler(Proxy.java:637) at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.cleanUp(BPOfferService.java:450) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.run(BPOfferService.java:639) at java.lang.Thread.run(Thread.java:662) 2011-12-17 15:56:26,248 ERROR ipc.RPC (RPC.java:stopProxy(559)) - Tried to call RPC.stopProxy on an object that is not a proxy. java.lang.IllegalArgumentException: not a proxy instance at java.lang.reflect.Proxy.getInvocationHandler(Proxy.java:637) at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.cleanUp(BPOfferService.java:450) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.run(BPOfferService.java:639) at java.lang.Thread.run(Thread.java:662)
        Hide
        Uma Maheswara Rao G added a comment -

        Here the problem is DatanodeProtocolClientSideTranslatorPB is not proxy instance directly. This is wrapper around the proxy instance of DatanodeProtocolPB.

        So, shutting down the previos cluster, the clinets are not getting cleared and they will be cached. When new cluster starts it may get old invalid clients and getting EOFEceptions.

        So, we should just call close of DatanodeProtocolClientSideTranslatorPB. That will call the RPC.stopProxy by passing the real proxy instance (DatanodeProtocolPB).

        Attached the patch by closing the DatanodeProtocolClientSideTranslatorPB.

        Thanks
        Uma

        Show
        Uma Maheswara Rao G added a comment - Here the problem is DatanodeProtocolClientSideTranslatorPB is not proxy instance directly. This is wrapper around the proxy instance of DatanodeProtocolPB. So, shutting down the previos cluster, the clinets are not getting cleared and they will be cached. When new cluster starts it may get old invalid clients and getting EOFEceptions. So, we should just call close of DatanodeProtocolClientSideTranslatorPB. That will call the RPC.stopProxy by passing the real proxy instance (DatanodeProtocolPB). Attached the patch by closing the DatanodeProtocolClientSideTranslatorPB. Thanks Uma
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12507785/HDFS-2700.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 90 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.fs.http.server.TestHttpFSServer
        org.apache.hadoop.lib.servlet.TestServerWebApp

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1722//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1722//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507785/HDFS-2700.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 90 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.http.server.TestHttpFSServer org.apache.hadoop.lib.servlet.TestServerWebApp +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1722//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1722//console This message is automatically generated.
        Hide
        Uma Maheswara Rao G added a comment -

        Test failures are unrelated to this patch. Already raised an issue for that failures HDFS-2657.
        Findbugs and javadoc comments also unrelated.

        Show
        Uma Maheswara Rao G added a comment - Test failures are unrelated to this patch. Already raised an issue for that failures HDFS-2657 . Findbugs and javadoc comments also unrelated.
        Hide
        Todd Lipcon added a comment -

        +1, thanks for taking care of this Uma. Will commit momentarily.

        Show
        Todd Lipcon added a comment - +1, thanks for taking care of this Uma. Will commit momentarily.
        Hide
        Todd Lipcon added a comment -

        Committed to trunk, thx Uma.

        Show
        Todd Lipcon added a comment - Committed to trunk, thx Uma.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1523 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1523/)
        HDFS-2700. Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1523 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1523/ ) HDFS-2700 . Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1450 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1450/)
        HDFS-2700. Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1450 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1450/ ) HDFS-2700 . Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1473 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1473/)
        HDFS-2700. Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1473 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1473/ ) HDFS-2700 . Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #898 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/898/)
        HDFS-2700. Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #898 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/898/ ) HDFS-2700 . Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #931 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/931/)
        HDFS-2700. Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #931 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/931/ ) HDFS-2700 . Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Hide
        Tsz Wo Nicholas Sze added a comment -

        TestDataNodeMultipleRegistrations is failing from last couple of builds
        https://builds.apache.org/job/PreCommit-HDFS-Build/lastCompletedBuild/testReport/

        It may be better to use the build number(s) instead of using the link with lastCompletedBuild next time since the "last" build keeps changing from time to time.

        Show
        Tsz Wo Nicholas Sze added a comment - TestDataNodeMultipleRegistrations is failing from last couple of builds https://builds.apache.org/job/PreCommit-HDFS-Build/lastCompletedBuild/testReport/ It may be better to use the build number(s) instead of using the link with lastCompletedBuild next time since the "last" build keeps changing from time to time.
        Hide
        Uma Maheswara Rao G added a comment -

        Sure, i will do that from next time.

        Show
        Uma Maheswara Rao G added a comment - Sure, i will do that from next time.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have merged this to 0.23.

        Show
        Tsz Wo Nicholas Sze added a comment - I have merged this to 0.23.

          People

          • Assignee:
            Uma Maheswara Rao G
            Reporter:
            Uma Maheswara Rao G
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development