Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4866

Protocol buffer support cannot compile under C

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.1.0-beta
    • Fix Version/s: 3.0.0, 2.1.0-beta
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      The Protocol Buffers definition of the inter-namenode protocol required a change for compatibility with compiled C clients. This is a backwards-incompatible change. A namenode prior to this change will not be able to communicate with a namenode after this change.
      Show
      The Protocol Buffers definition of the inter-namenode protocol required a change for compatibility with compiled C clients. This is a backwards-incompatible change. A namenode prior to this change will not be able to communicate with a namenode after this change.
    • Target Version/s:

      Description

      When compiling Hadoop's .proto descriptions for use in C, an error occurs because one of the RPC's in NamenodeProtocol.proto is named "register". This name is a reserved word in languages such as C. When using the Java and C++ languages, the name is hidden inside a class and therefore doesn't cause an error. Unfortunately, that is not the case in non-class languages such as C.

      Note: generating the C translation of the .proto files requires installation of the protobuf-c package from google:

      http://code.google.com/p/protobuf-c/

      1. pcreate.pl
        8 kB
        Ralph Castain
      2. HDFS-4866.002.patch
        6 kB
        Arpit Agarwal
      3. HDFS-4866.trunk.001.patch
        7 kB
        Arpit Agarwal
      4. HDFS-4866.branch-2.001.patch
        6 kB
        Arpit Agarwal
      5. pcreate.pl
        6 kB
        Ralph Castain
      6. NamenodeProtocol.pb-c.h
        46 kB
        Ralph Castain
      7. NamenodeProtocol.pb-c.c
        79 kB
        Ralph Castain

        Issue Links

          Activity

          Hide
          Ralph Castain added a comment -

          Details of the error when executing the protobuf-c parser on NamenodeProtocol.proto:

          protoc-c --c_out=. -I . -I<path-to-other-imports> NamenodeProtocol.proto

          The "register" function appears on line 714 of the resulting NamenodeProtocol.pb-c.h file:

            void (*register)(Hadoop__Hdfs__NamenodeProtocolService_Service *service,
                             const Hadoop__Hdfs__RegisterRequestProto *input,
                             Hadoop__Hdfs__RegisterResponseProto_Closure closure,
                             void *closure_data);
          

          It's a direct consequence of line 228 in the .proto file:

            /*
             * Request to register a sub-ordinate namenode
             */
            rpc register(RegisterRequestProto) returns(RegisterResponseProto);
          

          The rpc needs to be renamed to something non-reserved.

          Show
          Ralph Castain added a comment - Details of the error when executing the protobuf-c parser on NamenodeProtocol.proto: protoc-c --c_out=. -I . -I<path-to-other-imports> NamenodeProtocol.proto The "register" function appears on line 714 of the resulting NamenodeProtocol.pb-c.h file: void (*register)(Hadoop__Hdfs__NamenodeProtocolService_Service *service, const Hadoop__Hdfs__RegisterRequestProto *input, Hadoop__Hdfs__RegisterResponseProto_Closure closure, void *closure_data); It's a direct consequence of line 228 in the .proto file: /* * Request to register a sub-ordinate namenode */ rpc register(RegisterRequestProto) returns(RegisterResponseProto); The rpc needs to be renamed to something non-reserved.
          Hide
          Ralph Castain added a comment -

          Files output from protoc-c parser

          Show
          Ralph Castain added a comment - Files output from protoc-c parser
          Hide
          Steve Loughran added a comment -

          Moved to HDFS as it is an NN protocol issue.

          1. I hope it's not too late to fix this; protocols are sensitive.
          2. we should really get jenkins to do the proto-c builds too, to catch regressions here
          Show
          Steve Loughran added a comment - Moved to HDFS as it is an NN protocol issue. I hope it's not too late to fix this; protocols are sensitive. we should really get jenkins to do the proto-c builds too, to catch regressions here
          Hide
          Ralph Castain added a comment -

          Thanks! Sorry I put it in the wrong place - wasn't sure how things are organized.

          Show
          Ralph Castain added a comment - Thanks! Sorry I put it in the wrong place - wasn't sure how things are organized.
          Hide
          Ralph Castain added a comment -

          For ease or reproducing the problem, I've attached a perl script that cycles across the Hadoop tree and compiles all the .proto files using protobuf-c. This script is executed in the top-level directory and just takes --prefix=<dir-for-output> as its argument.

          Show
          Ralph Castain added a comment - For ease or reproducing the problem, I've attached a perl script that cycles across the Hadoop tree and compiles all the .proto files using protobuf-c. This script is executed in the top-level directory and just takes --prefix=<dir-for-output> as its argument.
          Hide
          Suresh Srinivas added a comment -

          This is a blocker for 2.1.0-beta.

          Show
          Suresh Srinivas added a comment - This is a blocker for 2.1.0-beta.
          Hide
          Arpit Agarwal added a comment -

          Rename register to registerSubordinateNamenode.

          Show
          Arpit Agarwal added a comment - Rename register to registerSubordinateNamenode.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12586801/HDFS-4866.trunk.001.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4499//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4499//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586801/HDFS-4866.trunk.001.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4499//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4499//console This message is automatically generated.
          Hide
          Arpit Agarwal added a comment -

          I believe no new tests are needed.

          Show
          Arpit Agarwal added a comment - I believe no new tests are needed.
          Hide
          Chris Nauroth added a comment -

          I've been trying to track down definitive documentation that tells us whether or not renaming a protobuf service RPC method is backwards-compatible at the protocol layer. Assume a client compiled before this change tried to connect to a server compiled after this change. Would this break, and if so, is there any alternative that achieves backwards-compatibility, or do we just need to accept the unfortunate backwards-incompatibility?

          I haven't found anything definitive in the protobuf documentation to answer my question. Have you found anything that discusses this?

          Show
          Chris Nauroth added a comment - I've been trying to track down definitive documentation that tells us whether or not renaming a protobuf service RPC method is backwards-compatible at the protocol layer. Assume a client compiled before this change tried to connect to a server compiled after this change. Would this break, and if so, is there any alternative that achieves backwards-compatibility, or do we just need to accept the unfortunate backwards-incompatibility? I haven't found anything definitive in the protobuf documentation to answer my question. Have you found anything that discusses this?
          Hide
          Arpit Agarwal added a comment -

          Hi Chris,

          Thanks for taking a look.

          I don't know of a way to avoid the backward incompatibility either.

          Note that this is the inter-namenode protocol so it should not affect older clients.

          Show
          Arpit Agarwal added a comment - Hi Chris, Thanks for taking a look. I don't know of a way to avoid the backward incompatibility either. Note that this is the inter-namenode protocol so it should not affect older clients.
          Hide
          Chris Nauroth added a comment -

          Recapping some offline discussion, the backwards-incompatibility is unavoidable. The RPC method name goes on the wire. Unfortunately, we have no choice but to make a backwards-incompatible change to enable a C client. At least the scope is limited to namenode protocol and not something more widely deployed like the client protocol.

          +1 for the patch. I plan to commit this and flag it as backwards-incompatible, but I'll wait one more day in case the other watchers on this issue want to offer additional feedback.

          Thinking ahead to future development using protobuf, we may want to declare a set of officially supported language bindings, take the union of all those languages' reserved words, and declare them off-limits in the proto specs. This could go into the code review checklist as a reminder.

          Show
          Chris Nauroth added a comment - Recapping some offline discussion, the backwards-incompatibility is unavoidable. The RPC method name goes on the wire. Unfortunately, we have no choice but to make a backwards-incompatible change to enable a C client. At least the scope is limited to namenode protocol and not something more widely deployed like the client protocol. +1 for the patch. I plan to commit this and flag it as backwards-incompatible, but I'll wait one more day in case the other watchers on this issue want to offer additional feedback. Thinking ahead to future development using protobuf, we may want to declare a set of officially supported language bindings, take the union of all those languages' reserved words, and declare them off-limits in the proto specs. This could go into the code review checklist as a reminder.
          Hide
          Arpit Agarwal added a comment -

          Rebasing patch.

          Show
          Arpit Agarwal added a comment - Rebasing patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12587900/HDFS-4866.002.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4521//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4521//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12587900/HDFS-4866.002.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4521//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4521//console This message is automatically generated.
          Hide
          Chris Nauroth added a comment -

          +1 for the rebased patch. I'm going to commit this.

          Show
          Chris Nauroth added a comment - +1 for the rebased patch. I'm going to commit this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3931 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3931/)
          HDFS-4866. Protocol buffer support cannot compile under C. Contributed by Arpit Agarwal. (Revision 1493300)

          Result = SUCCESS
          cnauroth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1493300
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocol.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3931 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3931/ ) HDFS-4866 . Protocol buffer support cannot compile under C. Contributed by Arpit Agarwal. (Revision 1493300) Result = SUCCESS cnauroth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1493300 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocol.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
          Hide
          Chris Nauroth added a comment -

          I committed this to trunk, branch-2, and branch-2.1-beta. Thanks to Arpit for coding the fix, and thanks to the numerous contributors who participated on discussion.

          Show
          Chris Nauroth added a comment - I committed this to trunk, branch-2, and branch-2.1-beta. Thanks to Arpit for coding the fix, and thanks to the numerous contributors who participated on discussion.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #241 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/241/)
          HDFS-4866. Protocol buffer support cannot compile under C. Contributed by Arpit Agarwal. (Revision 1493300)

          Result = SUCCESS
          cnauroth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1493300
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocol.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #241 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/241/ ) HDFS-4866 . Protocol buffer support cannot compile under C. Contributed by Arpit Agarwal. (Revision 1493300) Result = SUCCESS cnauroth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1493300 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocol.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1431 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1431/)
          HDFS-4866. Protocol buffer support cannot compile under C. Contributed by Arpit Agarwal. (Revision 1493300)

          Result = FAILURE
          cnauroth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1493300
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocol.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1431 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1431/ ) HDFS-4866 . Protocol buffer support cannot compile under C. Contributed by Arpit Agarwal. (Revision 1493300) Result = FAILURE cnauroth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1493300 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocol.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1458 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1458/)
          HDFS-4866. Protocol buffer support cannot compile under C. Contributed by Arpit Agarwal. (Revision 1493300)

          Result = FAILURE
          cnauroth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1493300
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocol.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1458 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1458/ ) HDFS-4866 . Protocol buffer support cannot compile under C. Contributed by Arpit Agarwal. (Revision 1493300) Result = FAILURE cnauroth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1493300 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocol.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
          Hide
          Ralph Castain added a comment -

          It seems to be compiling for me, which is a step forward. However, I still cannot construct a library of the results. This may be a Hadoop issue, or could be an issue with the protobuf-c cross-compiler. What I see are a bunch of these when attempting to link the resulting .o files:

          /home/common/hadoop/hadoop-common/foo/obj/DatanodeProtocol.pb-c.o: In function `hadoop_hdfsreport_bad_blocks_request_proto_init':
          DatanodeProtocol.pb-c.c.text+0x2bb4): multiple definition of `hadoop_hdfsreport_bad_blocks_request_proto_init'
          /home/common/hadoop/hadoop-common/foo/obj/ClientNamenodeProtocol.pb-c.o:ClientNamenodeProtocol.pb-c.c.text+0x277d): first defined here

          From what I can see, this is caused by the

          package hadoop.hdfs;

          line in the .proto files, when combined with the later

          import "hdfs.proto";

          This appears to bring a complete copy of the hdfs.proto file into the source code, which then recompiles it - leading to the duplicate symbols.

          I'll attach an updated pcreate.pl script that illustrates the problem. Excluding the following .proto files allows all to be successfully built and linked:

          DatanodeProtocol
          ClientNamenodeProtocol
          QJournalProtocol

          HTH
          Ralph

          Show
          Ralph Castain added a comment - It seems to be compiling for me, which is a step forward. However, I still cannot construct a library of the results. This may be a Hadoop issue, or could be an issue with the protobuf-c cross-compiler. What I see are a bunch of these when attempting to link the resulting .o files: /home/common/hadoop/hadoop-common/foo/obj/DatanodeProtocol.pb-c.o: In function `hadoop_ hdfs report_bad_blocks_request_proto _init': DatanodeProtocol.pb-c.c .text+0x2bb4): multiple definition of `hadoop_ hdfs report_bad_blocks_request_proto _init' /home/common/hadoop/hadoop-common/foo/obj/ClientNamenodeProtocol.pb-c.o:ClientNamenodeProtocol.pb-c.c .text+0x277d): first defined here From what I can see, this is caused by the package hadoop.hdfs; line in the .proto files, when combined with the later import "hdfs.proto"; This appears to bring a complete copy of the hdfs.proto file into the source code, which then recompiles it - leading to the duplicate symbols. I'll attach an updated pcreate.pl script that illustrates the problem. Excluding the following .proto files allows all to be successfully built and linked: DatanodeProtocol ClientNamenodeProtocol QJournalProtocol HTH Ralph
          Hide
          Ralph Castain added a comment -

          Updated version

          Show
          Ralph Castain added a comment - Updated version
          Hide
          Arpit Agarwal added a comment -

          Ralph,

          Can you please file a separate Jira for the link issue?

          Thanks,
          Arpit

          Show
          Arpit Agarwal added a comment - Ralph, Can you please file a separate Jira for the link issue? Thanks, Arpit

            People

            • Assignee:
              Arpit Agarwal
              Reporter:
              Ralph Castain
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development