Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2191

HDFS Minicluster tests failing after protobuf upgrade.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.5.0
    • Component/s: None
    • Labels:
      None

      Description

      I ran the full build in hadoop-1 profile, but it looks like the protobuf upgrade broke the hadoop-2 profile. The HDFS Sink test on Minicluster fails with this:

      Running org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster
      2013-09-13 12:11:31.159 java[58566:1203] Unable to load realm info from SCDynamicStore
      2013-09-13 12:11:31.208 java[58566:1203] Unable to load realm info from SCDynamicStore
      Tests run: 4, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 4.238 sec <<< FAILURE!
      simpleHDFSTest(org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster)  Time elapsed: 1979 sec  <<< ERROR!
      java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
      	at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
      	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetDatanodeReportRequestProto.getSerializedSize(ClientNamenodeProtocolProtos.java:21638)
      	at com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.constructRpcRequest(ProtobufRpcEngine.java:137)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:181)
      	at com.sun.proxy.$Proxy15.getDatanodeReport(Unknown Source)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:165)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:84)
      	at com.sun.proxy.$Proxy15.getDatanodeReport(Unknown Source)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDatanodeReport(ClientNamenodeProtocolTranslatorPB.java:488)
      	at org.apache.hadoop.hdfs.DFSClient.datanodeReport(DFSClient.java:1642)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:1703)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:1722)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1066)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:929)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:588)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:527)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:398)
      	at org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster.simpleHDFSTest(TestHDFSEventSinkOnMiniCluster.java:85)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
      	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
      	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
      	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
      	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
      
      
      1. FLUME-2191.patch
        1 kB
        Hari Shreedharan

        Issue Links

          Activity

          Hide
          hshreedharan Hari Shreedharan added a comment -

          Roshan Naik Any idea what is happening here?

          Show
          hshreedharan Hari Shreedharan added a comment - Roshan Naik Any idea what is happening here?
          Hide
          hshreedharan Hari Shreedharan added a comment -

          They hit similar issues in hadoop-2 profile, since Hadoop has code generated by protobuf-2.4.1 in their binaries.

          Show
          hshreedharan Hari Shreedharan added a comment - They hit similar issues in hadoop-2 profile, since Hadoop has code generated by protobuf-2.4.1 in their binaries.
          Hide
          hshreedharan Hari Shreedharan added a comment -

          Upgrades HDFS to 2.1.0-beta and includes the guava upgrade required by this version

          Show
          hshreedharan Hari Shreedharan added a comment - Upgrades HDFS to 2.1.0-beta and includes the guava upgrade required by this version
          Hide
          aprabhakar Arvind Prabhakar added a comment -

          +1 changes look good to me. Will commit after a sanity run.

          Show
          aprabhakar Arvind Prabhakar added a comment - +1 changes look good to me. Will commit after a sanity run.
          Hide
          aprabhakar Arvind Prabhakar added a comment -

          Patch committed to flume-1.5 and trunk. Thanks Hari!

          Show
          aprabhakar Arvind Prabhakar added a comment - Patch committed to flume-1.5 and trunk. Thanks Hari!
          Hide
          hudson Hudson added a comment -

          UNSTABLE: Integrated in flume-trunk #508 (See https://builds.apache.org/job/flume-trunk/508/)
          FLUME-2191. HDFS Minicluster tests failing after protobuf upgrade. (arvind: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9c59a309764498c013ccd202926d86413da01078)

          • flume-ng-sinks/flume-hdfs-sink/pom.xml
          • pom.xml
          Show
          hudson Hudson added a comment - UNSTABLE: Integrated in flume-trunk #508 (See https://builds.apache.org/job/flume-trunk/508/ ) FLUME-2191 . HDFS Minicluster tests failing after protobuf upgrade. (arvind: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9c59a309764498c013ccd202926d86413da01078 ) flume-ng-sinks/flume-hdfs-sink/pom.xml pom.xml

            People

            • Assignee:
              hshreedharan Hari Shreedharan
              Reporter:
              hshreedharan Hari Shreedharan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development