Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2700

TestDataNodeMultipleRegistrations is failing in trunk

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestDataNodeMultipleRegistrations is failing from last couple of builds
      https://builds.apache.org/job/PreCommit-HDFS-Build/lastCompletedBuild/testReport/

      1. HDFS-2700.patch
        1 kB
        Uma Maheswara Rao G

        Activity

        Uma Maheswara Rao G created issue -
        Hide
        Uma Maheswara Rao G added a comment -

        more info:
        java.io.IOException: Failed on local exception: java.io.EOFException; Host Details : local host is: "asf001.sp2.ygridcore.net/67.195.138.31"; destination host is: ""localhost":9929;
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:655)
        at org.apache.hadoop.ipc.Client.call(Client.java:1140)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:169)
        at $Proxy14.getDatanodeReport(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:127)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:81)
        at $Proxy14.getDatanodeReport(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDatanodeReport(ClientNamenodeProtocolTranslatorPB.java:555)
        at org.apache.hadoop.hdfs.DFSClient.datanodeReport(DFSClient.java:1443)
        at org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:1486)
        at org.apache.hadoop.hdfs.MiniDFSCluster.addNameNode(MiniDFSCluster.java:1904)
        at org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations.testMiniDFSClusterWithMultipleNN(TestDataNodeMultipleRegistrations.java:237)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        Show
        Uma Maheswara Rao G added a comment - more info: java.io.IOException: Failed on local exception: java.io.EOFException; Host Details : local host is: "asf001.sp2.ygridcore.net/67.195.138.31"; destination host is: ""localhost":9929; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:655) at org.apache.hadoop.ipc.Client.call(Client.java:1140) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:169) at $Proxy14.getDatanodeReport(Unknown Source) at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:127) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:81) at $Proxy14.getDatanodeReport(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDatanodeReport(ClientNamenodeProtocolTranslatorPB.java:555) at org.apache.hadoop.hdfs.DFSClient.datanodeReport(DFSClient.java:1443) at org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:1486) at org.apache.hadoop.hdfs.MiniDFSCluster.addNameNode(MiniDFSCluster.java:1904) at org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations.testMiniDFSClusterWithMultipleNN(TestDataNodeMultipleRegistrations.java:237) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        Hide
        Uma Maheswara Rao G added a comment -

        Some more information:

        Looks nor able clean the some of the proxy instances

        BP-239265342-67.195.138.31-1324137379885 (storage id DS-47228547-67.195.138.31-49285-1324137385405) registered with localhost/127.0.0.1:9930
        2011-12-17 15:56:26,248 ERROR ipc.RPC (RPC.java:stopProxy(559)) - Tried to call RPC.stopProxy on an object that is not a proxy.
        java.lang.IllegalArgumentException: not a proxy instance
        at java.lang.reflect.Proxy.getInvocationHandler(Proxy.java:637)
        at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.cleanUp(BPOfferService.java:450)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.run(BPOfferService.java:639)
        at java.lang.Thread.run(Thread.java:662)
        2011-12-17 15:56:26,248 ERROR ipc.RPC (RPC.java:stopProxy(559)) - Tried to call RPC.stopProxy on an object that is not a proxy.
        java.lang.IllegalArgumentException: not a proxy instance
        at java.lang.reflect.Proxy.getInvocationHandler(Proxy.java:637)
        at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.cleanUp(BPOfferService.java:450)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.run(BPOfferService.java:639)
        at java.lang.Thread.run(Thread.java:662)

        Show
        Uma Maheswara Rao G added a comment - Some more information: Looks nor able clean the some of the proxy instances BP-239265342-67.195.138.31-1324137379885 (storage id DS-47228547-67.195.138.31-49285-1324137385405) registered with localhost/127.0.0.1:9930 2011-12-17 15:56:26,248 ERROR ipc.RPC (RPC.java:stopProxy(559)) - Tried to call RPC.stopProxy on an object that is not a proxy. java.lang.IllegalArgumentException: not a proxy instance at java.lang.reflect.Proxy.getInvocationHandler(Proxy.java:637) at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.cleanUp(BPOfferService.java:450) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.run(BPOfferService.java:639) at java.lang.Thread.run(Thread.java:662) 2011-12-17 15:56:26,248 ERROR ipc.RPC (RPC.java:stopProxy(559)) - Tried to call RPC.stopProxy on an object that is not a proxy. java.lang.IllegalArgumentException: not a proxy instance at java.lang.reflect.Proxy.getInvocationHandler(Proxy.java:637) at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.cleanUp(BPOfferService.java:450) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.run(BPOfferService.java:639) at java.lang.Thread.run(Thread.java:662)
        Hide
        Uma Maheswara Rao G added a comment -

        Here the problem is DatanodeProtocolClientSideTranslatorPB is not proxy instance directly. This is wrapper around the proxy instance of DatanodeProtocolPB.

        So, shutting down the previos cluster, the clinets are not getting cleared and they will be cached. When new cluster starts it may get old invalid clients and getting EOFEceptions.

        So, we should just call close of DatanodeProtocolClientSideTranslatorPB. That will call the RPC.stopProxy by passing the real proxy instance (DatanodeProtocolPB).

        Attached the patch by closing the DatanodeProtocolClientSideTranslatorPB.

        Thanks
        Uma

        Show
        Uma Maheswara Rao G added a comment - Here the problem is DatanodeProtocolClientSideTranslatorPB is not proxy instance directly. This is wrapper around the proxy instance of DatanodeProtocolPB. So, shutting down the previos cluster, the clinets are not getting cleared and they will be cached. When new cluster starts it may get old invalid clients and getting EOFEceptions. So, we should just call close of DatanodeProtocolClientSideTranslatorPB. That will call the RPC.stopProxy by passing the real proxy instance (DatanodeProtocolPB). Attached the patch by closing the DatanodeProtocolClientSideTranslatorPB. Thanks Uma
        Uma Maheswara Rao G made changes -
        Field Original Value New Value
        Attachment HDFS-2700.patch [ 12507785 ]
        Uma Maheswara Rao G made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Uma Maheswara Rao G [ umamaheswararao ]
        Uma Maheswara Rao G made changes -
        Issue Type Test [ 6 ] Bug [ 1 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12507785/HDFS-2700.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 90 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.fs.http.server.TestHttpFSServer
        org.apache.hadoop.lib.servlet.TestServerWebApp

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1722//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1722//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507785/HDFS-2700.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 90 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.http.server.TestHttpFSServer org.apache.hadoop.lib.servlet.TestServerWebApp +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1722//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1722//console This message is automatically generated.
        Hide
        Uma Maheswara Rao G added a comment -

        Test failures are unrelated to this patch. Already raised an issue for that failures HDFS-2657.
        Findbugs and javadoc comments also unrelated.

        Show
        Uma Maheswara Rao G added a comment - Test failures are unrelated to this patch. Already raised an issue for that failures HDFS-2657 . Findbugs and javadoc comments also unrelated.
        Hide
        Todd Lipcon added a comment -

        +1, thanks for taking care of this Uma. Will commit momentarily.

        Show
        Todd Lipcon added a comment - +1, thanks for taking care of this Uma. Will commit momentarily.
        Hide
        Todd Lipcon added a comment -

        Committed to trunk, thx Uma.

        Show
        Todd Lipcon added a comment - Committed to trunk, thx Uma.
        Todd Lipcon made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 0.24.0 [ 12317653 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1523 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1523/)
        HDFS-2700. Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1523 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1523/ ) HDFS-2700 . Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1450 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1450/)
        HDFS-2700. Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1450 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1450/ ) HDFS-2700 . Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1473 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1473/)
        HDFS-2700. Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1473 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1473/ ) HDFS-2700 . Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #898 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/898/)
        HDFS-2700. Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #898 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/898/ ) HDFS-2700 . Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #931 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/931/)
        HDFS-2700. Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #931 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/931/ ) HDFS-2700 . Fix failing TestDataNodeMultipleRegistrations in trunk. Contributed by Uma Maheswara Rao G. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220315 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        Hide
        Tsz Wo Nicholas Sze added a comment -

        TestDataNodeMultipleRegistrations is failing from last couple of builds
        https://builds.apache.org/job/PreCommit-HDFS-Build/lastCompletedBuild/testReport/

        It may be better to use the build number(s) instead of using the link with lastCompletedBuild next time since the "last" build keeps changing from time to time.

        Show
        Tsz Wo Nicholas Sze added a comment - TestDataNodeMultipleRegistrations is failing from last couple of builds https://builds.apache.org/job/PreCommit-HDFS-Build/lastCompletedBuild/testReport/ It may be better to use the build number(s) instead of using the link with lastCompletedBuild next time since the "last" build keeps changing from time to time.
        Hide
        Uma Maheswara Rao G added a comment -

        Sure, i will do that from next time.

        Show
        Uma Maheswara Rao G added a comment - Sure, i will do that from next time.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have merged this to 0.23.

        Show
        Tsz Wo Nicholas Sze added a comment - I have merged this to 0.23.
        Tsz Wo Nicholas Sze made changes -
        Fix Version/s 0.23.3 [ 12320052 ]
        Arun C Murthy made changes -
        Fix Version/s 2.0.0 [ 12320353 ]
        Fix Version/s 0.24.0 [ 12317653 ]
        Fix Version/s 0.23.3 [ 12320052 ]
        Vinod Kumar Vavilapalli made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        1h 22m 1 Uma Maheswara Rao G 17/Dec/11 17:45
        Patch Available Patch Available Resolved Resolved
        7h 25m 1 Todd Lipcon 18/Dec/11 01:11
        Resolved Resolved Closed Closed
        1380d 19h 47m 1 Vinod Kumar Vavilapalli 28/Sep/15 20:58

          People

          • Assignee:
            Uma Maheswara Rao G
            Reporter:
            Uma Maheswara Rao G
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development