Hadoop Common
  1. Hadoop Common
  2. HADOOP-9346

Upgrading to protoc 2.5.0 fails the build

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Duplicate
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: build
    • Labels:

      Description

      Reported over the impala lists, one of the errors received is:

      src/hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ha/proto/ZKFCProtocolProtos.java:[104,37] can not find symbol.
      symbol: class Parser
      location: package com.google.protobuf
      

      Worth looking into as we'll eventually someday bump our protobuf deps.

        Issue Links

          Activity

          Hide
          Harsh J added a comment -

          Thanks for pinging Ravi. I'd discussed with Alejandro that this could be closed. Looks like we added a dupe link but failed to close. Closing now.

          Show
          Harsh J added a comment - Thanks for pinging Ravi. I'd discussed with Alejandro that this could be closed. Looks like we added a dupe link but failed to close. Closing now.
          Hide
          Ravi Prakash added a comment -

          Can we close this JIRA now? Now that HADOOP-9845 is in trunk?

          Show
          Ravi Prakash added a comment - Can we close this JIRA now? Now that HADOOP-9845 is in trunk?
          Hide
          Tian Hong Wang added a comment -

          hi Harsh, When using protoc2.5.0, I met the problem of HADOOP-9440. It seems that protoc2.5.0 add new code.

          Show
          Tian Hong Wang added a comment - hi Harsh, When using protoc2.5.0, I met the problem of HADOOP-9440 . It seems that protoc2.5.0 add new code.
          Hide
          Harsh J added a comment -

          Cancelling patch while we address Ravi's find (and any other such subtle issues that tests are seemingly not capturing)

          Show
          Harsh J added a comment - Cancelling patch while we address Ravi's find (and any other such subtle issues that tests are seemingly not capturing)
          Hide
          Harsh J added a comment -

          Thanks Ravi!

          No I did not actually build a package and run the daemons. The error does look protobuf related. I'll investigate!

          Show
          Harsh J added a comment - Thanks Ravi! No I did not actually build a package and run the daemons. The error does look protobuf related. I'll investigate!
          Hide
          Ravi Prakash added a comment -

          Hi Harsh. Thanks. This is a thorn in my side as well.

          The code builds and I tried wordcount on Fedora 19 + protobuf-2.5.

          However on Fedora 19 + protobuf 2.4 even though the build succeeds with the patch, the DN doesn't come up:

          2013-05-14 14:26:30,916 [DataNode: file:/tmp/hadoop-raviprak/hdfs/data1,file:/tmp/hadoop-raviprak/hdfs/data2 heartbeating to NAMENODE-HOSTNAME/NAMENODE-IP:9000] WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected exception in block pool Block pool <registering> (storage id unknown) service to NAMENODE-HOSTNAME/NAMENODE-IP:9000
          java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
          at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
          at org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$VersionRequestProto.getSerializedSize(HdfsProtos.java:20296)
          at com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49)
          at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.constructRpcRequest(ProtobufRpcEngine.java:149)
          at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:193)
          at com.sun.proxy.$Proxy9.versionRequest(Unknown Source)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:601)
          at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
          at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
          at com.sun.proxy.$Proxy9.versionRequest(Unknown Source)
          at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.versionRequest(DatanodeProtocolClientSideTranslatorPB.java:245)
          at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.retrieveNamespaceInfo(BPServiceActor.java:163)
          at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:217)
          at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
          at java.lang.Thread.run(Thread.java:722)

          Were you able to get the daemon up? Is this something wrong with my environment? Without the patch the daemon came up and I was able to run a wordcount

          Show
          Ravi Prakash added a comment - Hi Harsh. Thanks. This is a thorn in my side as well. The code builds and I tried wordcount on Fedora 19 + protobuf-2.5. However on Fedora 19 + protobuf 2.4 even though the build succeeds with the patch, the DN doesn't come up: 2013-05-14 14:26:30,916 [DataNode: file:/tmp/hadoop-raviprak/hdfs/data1,file:/tmp/hadoop-raviprak/hdfs/data2 heartbeating to NAMENODE-HOSTNAME/NAMENODE-IP:9000] WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected exception in block pool Block pool <registering> (storage id unknown) service to NAMENODE-HOSTNAME/NAMENODE-IP:9000 java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses. at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180) at org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$VersionRequestProto.getSerializedSize(HdfsProtos.java:20296) at com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.constructRpcRequest(ProtobufRpcEngine.java:149) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:193) at com.sun.proxy.$Proxy9.versionRequest(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83) at com.sun.proxy.$Proxy9.versionRequest(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.versionRequest(DatanodeProtocolClientSideTranslatorPB.java:245) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.retrieveNamespaceInfo(BPServiceActor.java:163) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:217) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664) at java.lang.Thread.run(Thread.java:722) Were you able to get the daemon up? Is this something wrong with my environment? Without the patch the daemon came up and I was able to run a wordcount
          Hide
          Harsh J added a comment -

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          The build passing is the test for this change - as the bin/protoc on the current build machines are that of a lower version than the bumped jar dependency.

          Besides that, both 2.5 and 2.4 protoc builds have been locally tested by me manually.

          Show
          Harsh J added a comment - -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. The build passing is the test for this change - as the bin/protoc on the current build machines are that of a lower version than the bumped jar dependency. Besides that, both 2.5 and 2.4 protoc builds have been locally tested by me manually.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12582893/HADOOP-9346.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2537//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2537//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582893/HADOOP-9346.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2537//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2537//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Seems like we can safely change this. I stood up a box with protoc --version == 2.4.1, applied this patch, did not upgrade protoc to 2.5, ran the build, it succeeds reliably.

          This patch should help users who use 2.5 protoc locally while not breaking those that are on 2.4.

          Show
          Harsh J added a comment - Seems like we can safely change this. I stood up a box with protoc --version == 2.4.1, applied this patch, did not upgrade protoc to 2.5, ran the build, it succeeds reliably. This patch should help users who use 2.5 protoc locally while not breaking those that are on 2.4.
          Hide
          Harsh J added a comment -

          The fix is expectedly simple (and is attached). The issue is that the protoc of 2.5.0 adds new calls/classes to generated code that the older 2.4.x jars do not carry. Since the project explicitly downloads version 2.4.0a, we run into various 'missing' form of errors.

          Show
          Harsh J added a comment - The fix is expectedly simple (and is attached). The issue is that the protoc of 2.5.0 adds new calls/classes to generated code that the older 2.4.x jars do not carry. Since the project explicitly downloads version 2.4.0a, we run into various 'missing' form of errors.
          Hide
          Ozzie Zhang added a comment -

          whether anyone solved this issue? when I build hadoop in the three different environments(mac, centos, debian), all results fail in this issue.

          my steps:

          1) git clone https://github.com/apache/hadoop-common.git

          2) download http://protobuf.googlecode.com/files/protobuf-2.5.0.tar.bz2 and install

          3) run "mvn package -Pdist,native,docs -DskipTests -Dtar"

          Show
          Ozzie Zhang added a comment - whether anyone solved this issue? when I build hadoop in the three different environments(mac, centos, debian), all results fail in this issue. my steps: 1) git clone https://github.com/apache/hadoop-common.git 2) download http://protobuf.googlecode.com/files/protobuf-2.5.0.tar.bz2 and install 3) run "mvn package -Pdist,native,docs -DskipTests -Dtar"

            People

            • Assignee:
              Harsh J
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development