Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-851

Hello-samza doesn't work on CDH

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0, 0.10.1
    • Fix Version/s: 0.10.0, 0.10.1
    • Component/s: hello-samza
    • Labels:
      None

      Description

      Hello-samza 0.10.0 doesn't work on CDH (tested on CDH5.4.0) due to incompatible Hadoop jars.

      It runs fine on local grid. But when running on cluster I can see it talks fine to YARN's ResourceManager and Kafka, get's resources, but fails to start container. At first, I thought it's because CDH5.4 uses YARN 2.6.0, while Samza uses 2.6.1, so I recompiled Samza with YARN 2.6.0 and re-assambled hello-samza. No luck so far. Also, I thought it's because of different protobuf version, but I checked, it's the same in CDH and Samza.

      Followed http://samza.apache.org/learn/tutorials/0.10/deploy-samza-to-CDH.html to deploy.

      See full logs:
      Samza Deploy Script output http://pastebin.com/0K8HVMzt
      YARN logs http://pastebin.com/UvNp6240

      An error is:
      java version "1.7.0_80"
      Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
      Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
      com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
      at com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
      at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
      at org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1078)
      at org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1042)
      at org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1138)
      at org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1133)
      at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
      at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
      at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
      at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
      at org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.parseFrom(YarnSecurityTokenProtos.java:1307)
      at org.apache.hadoop.yarn.security.AMRMTokenIdentifier.readFields(AMRMTokenIdentifier.java:81)
      at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:142)
      at org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:181)
      at org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
      at org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
      at java.lang.Thread.run(Thread.java:745)
      Exception in thread "Container Allocator Thread" org.apache.samza.SamzaException: IO Exception when writing credentials to output buffer
      at org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:190)
      at org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
      at org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
      at java.lang.Thread.run(Thread.java:745)

        Attachments

        1. 0001-Maven-profile-for-the-CDH-compatible-build-replacing.patch
          3 kB
          Stanislav Los
        2. SAMZA-851-0-doc.patch
          1 kB
          Yi Pan
        3. SAMZA-851-latest-pom.patch
          3 kB
          Yi Pan
        4. SAMZA-851-master-pom.patch
          3 kB
          Yi Pan
        5. SAMZA-851-doc-0.10.0.patch
          1 kB
          Yi Pan

          Activity

            People

            • Assignee:
              staslos Stanislav Los
              Reporter:
              staslos Stanislav Los
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: