Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.24.0
    • Fix Version/s: 2.0.0-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Built a 0.24-SNAPSHOT tar from today, used a general config, started NN/DN, but SNN won't come up with following error:

      11/12/31 12:13:14 ERROR namenode.SecondaryNameNode: Throwable Exception in doCheckpoint
      java.lang.RuntimeException: java.lang.NoSuchFieldException: versionID
      	at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:154)
      	at org.apache.hadoop.ipc.WritableRpcEngine$Invocation.<init>(WritableRpcEngine.java:112)
      	at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:226)
      	at $Proxy9.getTransationId(Unknown Source)
      	at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.getTransactionID(NamenodeProtocolTranslatorPB.java:185)
      	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.countUncheckpointedTxns(SecondaryNameNode.java:625)
      	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.shouldCheckpointBasedOnCount(SecondaryNameNode.java:633)
      	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:386)
      	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:356)
      	at java.lang.Thread.run(Thread.java:680)
      Caused by: java.lang.NoSuchFieldException: versionID
      	at java.lang.Class.getField(Class.java:1520)
      	at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:150)
      	... 9 more
      java.lang.RuntimeException: java.lang.NoSuchFieldException: versionID
      	at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:154)
      	at org.apache.hadoop.ipc.WritableRpcEngine$Invocation.<init>(WritableRpcEngine.java:112)
      	at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:226)
      	at $Proxy9.getTransationId(Unknown Source)
      	at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.getTransactionID(NamenodeProtocolTranslatorPB.java:185)
      	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.countUncheckpointedTxns(SecondaryNameNode.java:625)
      	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.shouldCheckpointBasedOnCount(SecondaryNameNode.java:633)
      	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:386)
      	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:356)
      	at java.lang.Thread.run(Thread.java:680)
      Caused by: java.lang.NoSuchFieldException: versionID
      	at java.lang.Class.getField(Class.java:1520)
      	at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:150)
      	... 9 more
      11/12/31 12:13:14 INFO namenode.SecondaryNameNode: SHUTDOWN_MSG: 
      /************************************************************
      SHUTDOWN_MSG: Shutting down SecondaryNameNode at sho-mba.local/192.168.11.2
      ************************************************************/
      

      full error log: http://pastebin.com/mSaVbS34

      1. HDFS-2739.trunk.patch
        9 kB
        Jitendra Nath Pandey

        Activity

        Sho Shimauchi created issue -
        Hide
        Harsh J added a comment -

        Also looks like a protocol or call somewhere seems to have a typo in it, i.e. getTransationId. We could fix it as part of this itself perhaps.

        Show
        Harsh J added a comment - Also looks like a protocol or call somewhere seems to have a typo in it, i.e. getTransationId . We could fix it as part of this itself perhaps.
        Harsh J made changes -
        Field Original Value New Value
        Affects Version/s 0.24.0 [ 12317653 ]
        Suresh Srinivas made changes -
        Assignee Suresh Srinivas [ sureshms ]
        Hide
        Jitendra Nath Pandey added a comment -

        The problem is that WritableRpc proxy is getting created in the PB translator. We don't see this problem in unit tests because rpc engines are globally configured.

        I have tested the patch on a single node installation.

        Also fixed the typo pointed out by Harsh.

        Show
        Jitendra Nath Pandey added a comment - The problem is that WritableRpc proxy is getting created in the PB translator. We don't see this problem in unit tests because rpc engines are globally configured. I have tested the patch on a single node installation. Also fixed the typo pointed out by Harsh.
        Jitendra Nath Pandey made changes -
        Attachment HDFS-2739.trunk.patch [ 12510134 ]
        Hide
        Suresh Srinivas added a comment -

        +1 for the patch.

        Show
        Suresh Srinivas added a comment - +1 for the patch.
        Jitendra Nath Pandey made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hadoop Flags Reviewed [ 10343 ]
        Assignee Suresh Srinivas [ sureshms ] Jitendra Nath Pandey [ jnp ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12510134/HDFS-2739.trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 21 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1771//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1771//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1771//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1771//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510134/HDFS-2739.trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 21 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1771//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1771//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1771//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1771//console This message is automatically generated.
        Hide
        Jitendra Nath Pandey added a comment -

        TestDistributedUpgrade failure is unrelated and it actually passes on my machine.
        The findbugs, release audit and javadoc warnings also seem to exist in trunk for a while now.

        I tested the patch manually because secondary namenode was required to be run as a separate process to reproduce the problem.

        Show
        Jitendra Nath Pandey added a comment - TestDistributedUpgrade failure is unrelated and it actually passes on my machine. The findbugs, release audit and javadoc warnings also seem to exist in trunk for a while now. I tested the patch manually because secondary namenode was required to be run as a separate process to reproduce the problem.
        Hide
        Jitendra Nath Pandey added a comment -

        Committed.

        Show
        Jitendra Nath Pandey added a comment - Committed.
        Jitendra Nath Pandey made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1597 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1597/)
        HDFS-2739. SecondaryNameNode doesn't start up.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229877
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1597 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1597/ ) HDFS-2739 . SecondaryNameNode doesn't start up. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229877 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1524 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1524/)
        HDFS-2739. SecondaryNameNode doesn't start up.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229877
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1524 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1524/ ) HDFS-2739 . SecondaryNameNode doesn't start up. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229877 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1543 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1543/)
        HDFS-2739. SecondaryNameNode doesn't start up.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229877
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1543 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1543/ ) HDFS-2739 . SecondaryNameNode doesn't start up. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229877 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #922 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/922/)
        HDFS-2739. SecondaryNameNode doesn't start up.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229877
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #922 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/922/ ) HDFS-2739 . SecondaryNameNode doesn't start up. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229877 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #955 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/955/)
        HDFS-2739. SecondaryNameNode doesn't start up.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229877
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #955 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/955/ ) HDFS-2739 . SecondaryNameNode doesn't start up. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229877 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have merged this to 0.23.

        Show
        Tsz Wo Nicholas Sze added a comment - I have merged this to 0.23.
        Tsz Wo Nicholas Sze made changes -
        Fix Version/s 0.23.3 [ 12320052 ]
        Fix Version/s 0.24.0 [ 12317653 ]
        Arun C Murthy made changes -
        Fix Version/s 2.0.0 [ 12320353 ]
        Fix Version/s 0.24.0 [ 12317653 ]
        Fix Version/s 0.23.3 [ 12320052 ]

          People

          • Assignee:
            Jitendra Nath Pandey
            Reporter:
            Sho Shimauchi
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development