Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2172

Update protocol buffer from 2.4.1 to 2.5.0

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.5.0
    • Component/s: Sinks+Sources
    • Labels:

      Description

      Hadoop and HBase have upgraded to 2.5.0 of protocol buffer.
      See HADOOP-9845

      Due to backward incompatibility, this introduces runtime errors in HDFS sink & HBase sink tests:

      class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$DeleteSnapshotResponseProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
      Stacktrace
      
      java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$DeleteSnapshotResponseProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
      	at java.lang.ClassLoader.defineClass1(Native Method)
      	at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
      	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
      	at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
      	at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.<clinit>(ClientNamenodeProtocolServerSideTranslatorPB.java:169)
      	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.<init>(NameNodeRpcServer.java:181)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:485)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:468)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:659)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:644)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1221)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:893)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:784)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:642)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:585)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:456)
      	at org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster.simpleHDFSTest(TestHDFSEventSinkOnMiniCluster.java:85)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
      	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
      	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
      	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
      	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
      
      
      1. FLUME-2172.patch
        289 kB
        Roshan Naik

        Activity

        Hide
        roshan_naik Roshan Naik added a comment -

        Upping the protocol buffer version

        Show
        roshan_naik Roshan Naik added a comment - Upping the protocol buffer version
        Hide
        roshan_naik Roshan Naik added a comment -

        Revised patch also contains protoc compiler generated code for file channel.

        Show
        roshan_naik Roshan Naik added a comment - Revised patch also contains protoc compiler generated code for file channel.
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Roshan - Did you verify that the data written by a protobuf-2.4.1 file channel can be read by the new code?

        Show
        hshreedharan Hari Shreedharan added a comment - Roshan - Did you verify that the data written by a protobuf-2.4.1 file channel can be read by the new code?
        Hide
        hshreedharan Hari Shreedharan added a comment -

        +1. Ran full build. Tested against pre-2.5.0 file channel.

        Show
        hshreedharan Hari Shreedharan added a comment - +1. Ran full build. Tested against pre-2.5.0 file channel.
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Committed, rev: 39b6b52. Thanks Roshan!

        Show
        hshreedharan Hari Shreedharan added a comment - Committed, rev: 39b6b52. Thanks Roshan!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in flume-trunk #493 (See https://builds.apache.org/job/flume-trunk/493/)
        FLUME-2172. Update protocol buffer from 2.4.1 to 2.5.0 (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=39b6b5282938b9429cee3acaac0763cedd8fa9f0)

        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java
        • pom.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in flume-trunk #493 (See https://builds.apache.org/job/flume-trunk/493/ ) FLUME-2172 . Update protocol buffer from 2.4.1 to 2.5.0 (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=39b6b5282938b9429cee3acaac0763cedd8fa9f0 ) flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java pom.xml
        Hide
        roshan_naik Roshan Naik added a comment -

        rebased patch

        Show
        roshan_naik Roshan Naik added a comment - rebased patch
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Roshan Naik - This was committed a while ago.

        Show
        hshreedharan Hari Shreedharan added a comment - Roshan Naik - This was committed a while ago.
        Hide
        roshan_naik Roshan Naik added a comment -

        yes my bad.. i was accidentally on the wrong jira. was intending to update another jira. i have deleted the revised patch form this jira.

        Show
        roshan_naik Roshan Naik added a comment - yes my bad.. i was accidentally on the wrong jira. was intending to update another jira. i have deleted the revised patch form this jira.
        Hide
        stanley_shi stanley shi added a comment -

        Just wondering: the attached patch is in-correct, why is this JIRA Resolved?

        Show
        stanley_shi stanley shi added a comment - Just wondering: the attached patch is in-correct, why is this JIRA Resolved?
        Hide
        gopinathan.av Gopinathan A added a comment -

        stanley shi proper patch (FLUME-2172.patch) got committed in trunk.

        I guess you confused with FLUME-2172.2.patch, this patch accidentally got added by Roshan.

        Please let us know if any other issues in FLUME-2172.patch, we can re-open this bug. We already ported this patch in our code base, its working fine.

        Show
        gopinathan.av Gopinathan A added a comment - stanley shi proper patch ( FLUME-2172 .patch) got committed in trunk. I guess you confused with FLUME-2172 .2.patch, this patch accidentally got added by Roshan. Please let us know if any other issues in FLUME-2172 .patch, we can re-open this bug. We already ported this patch in our code base, its working fine.
        Hide
        stanley_shi stanley shi added a comment -

        Thanks [~agopinath], I thought the newer patch should be the right patch.

        Show
        stanley_shi stanley shi added a comment - Thanks [~agopinath] , I thought the newer patch should be the right patch.
        Hide
        roshan_naik Roshan Naik added a comment -

        i meant to delete the erroneous patch but didnt. Just deleted it. sorry for the confusion.

        Show
        roshan_naik Roshan Naik added a comment - i meant to delete the erroneous patch but didnt. Just deleted it. sorry for the confusion.

          People

          • Assignee:
            roshan_naik Roshan Naik
            Reporter:
            roshan_naik Roshan Naik
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development