Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1981

Serialization error with TimerInternals in ApexGroupByKeyOperator

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.0.0
    • runner-apex
    • None

    Description

      Logs below. We tried switching to Java serialization, but that didn't work. We made the field transient (which is broken but let us make progress) and that did.

      Stack trace

      com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
      Serialization trace:
      activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
      	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
      	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
      	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
      	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
      	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
      	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
      	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
      	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
       context: PTContainer[id=6(container_1492195730173_0001_01_000012),state=ACTIVE,operators=[
      

      Larger logs with more scope:

      	
      2017-04-14 18:56:49,961 INFO com.datatorrent.stram.StreamingAppMaster: Master starting with classpath: ./portability-demo-bundled-apex.jar:/etc/hadoop/conf:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-auth-2.7.3.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-common-2.7.3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7.3.jar:/usr/lib/hadoop/hadoop-nfs-2.7.3.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/hadoop-common-2.7.3.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/snappy-java-1.0.5.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/hadoop-lzo-0.4.19.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/bigquery-connector-0.10.1-hadoop2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/avro-1.7.7.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/gcs-connector-1.6.0-hadoop2.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/zookeeper-3.4.6.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jetty-6.1.26.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/httpclient-4.2.5.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/httpcore-4.2.5.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.5.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/hadoop-auth-2.7.3.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-1.7.4.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7.3.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/hadoop-extras-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-aws-2.7.3.jar:/usr/lib/hadoop-mapreduce/avro-1.7.7.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar:/usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-archives.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/zookeeper-3.4.6.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26.jar:/usr/lib/hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/joda-time-2.9.9.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.42.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix-2.7.3.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/usr/lib/hadoop-mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.7.3.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/httpclient-4.2.5.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar:/usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/jackson-annotations-2.2.3.jar:/usr/lib/hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/jackson-databind-2.2.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.7.3.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/httpcore-4.2.5.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3.jar:/usr/lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop-mapreduce/hadoop-aws.jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/jackson-core-2.2.3.jar:/usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/usr/lib/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.5.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.7.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/lib/*'
      2017-04-14 18:56:49,966 INFO com.datatorrent.stram.StreamingAppMaster: version: 3.5.0 from rev: 6de8828 branch: 6de8828e4f3d5734d0a6f9c1be0aa7057cb60ac8 by Thomas Weise on 06.12.2016 @ 22:11:51 PST
      2017-04-14 18:56:49,967 INFO com.datatorrent.stram.StreamingAppMaster: appmaster env:
      PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
      HADOOP_CONF_DIR=/etc/hadoop/conf
      MAX_APP_ATTEMPTS=2
      DATAPROC_MASTER_STANDALONE_COMPONENTS=hadoop-hdfs-secondarynamenode
      DATAPROC_COMMON_COMPONENTS=openjdk-8-jdk libjansi-java python-numpy libmysql-java hadoop-client hive pig spark-core spark-python spark-r autofs nfs-common libhdfs0 libsnappy1 libatlas3-base libopenblas-base libapr1 vim git bash-completion spark-yarn-shuffle spark-datanucleus spark-extras hadoop-lzo
      MAIL=/var/mail/yarn
      LD_LIBRARY_PATH=:/usr/lib/hadoop/lib/native:/usr/lib/hadoop/lib/native
      HADOOP_HOME_WARN_SUPPRESS=true
      DATAPROC_WORKER_COMPONENTS=hadoop-hdfs-datanode hadoop-yarn-nodemanager
      LOGNAME=dhalperi
      JVM_PID=2123
      JSVC_HOME=/usr/lib/bigtop-utils
      PWD=/hadoop/yarn/nm-local-dir/usercache/dhalperi/appcache/application_1492195730173_0001/container_1492195730173_0001_01_000001
      HADOOP_YARN_USER=yarn
      HADOOP_PREFIX=/usr/lib/hadoop
      LOCAL_DIRS=/hadoop/yarn/nm-local-dir/usercache/dhalperi/appcache/application_1492195730173_0001
      YARN_IDENT_STRING=yarn
      SHELL=/bin/bash
      ALPN_JAR=/usr/local/share/google/alpn/alpn-boot-8.1.7.v20160121.jar
      YARN_CONF_DIR=/etc/hadoop/conf
      _SYSTEMCTL_SKIP_REDIRECT=true
      LOG_DIRS=/var/log/hadoop-yarn/userlogs/application_1492195730173_0001/container_1492195730173_0001_01_000001
      NM_AUX_SERVICE_mapreduce_shuffle=AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
      
      BDUTIL_DIR=/usr/local/share/google/dataproc/bdutil-dataproc-20170410-124434-RC1
      HADOOP_YARN_HOME=/usr/lib/hadoop-yarn
      YARN_PID_DIR=/var/run/hadoop-yarn
      DATAPROC_MASTER_EXCLUSIVE_COMPONENTS=hadoop-mapreduce-historyserver hive-metastore hive-server2 nfs-kernel-server spark-history-server
      NM_AUX_SERVICE_spark_shuffle=
      HADOOP_HOME=/usr/lib/hadoop
      SHLVL=5
      YARN_ROOT_LOGGER=INFO,RFA
      JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
      LANG=en_US.UTF-8
      APP_SUBMIT_TIME_ENV=1492196207488
      NM_HOST=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal
      YARN_LOGFILE=yarn-yarn-nodemanager-gaming-apex-dhalperi-w-15.log
      HADOOP_USER_NAME=dhalperi
      DATAPROC_AGENT_JAR=/usr/local/share/google/dataproc/agent-20170410-124434-RC1.jar
      HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec
      YARN_LOG_DIR=/var/log/hadoop-yarn
      HADOOP_HDFS_HOME=/usr/lib/hadoop-hdfs
      HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce
      HADOOP_COMMON_HOME=/usr/lib/hadoop
      _=/usr/lib/jvm/java-8-openjdk-amd64/bin/java
      APPLICATION_WEB_PROXY_BASE=/proxy/application_1492195730173_0001
      NM_HTTP_PORT=8042
      NM_PORT=53054
      USER=dhalperi
      CLASSPATH=./*:/etc/hadoop/conf:/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-yarn/*:/usr/lib/hadoop-yarn/lib/*'
      DATAPROC_MASTER_HA_COMPONENTS=hadoop-hdfs-journalnode hadoop-hdfs-zkfc zookeeper-server
      HADOOP_TOKEN_FILE_LOCATION=/hadoop/yarn/nm-local-dir/usercache/dhalperi/appcache/application_1492195730173_0001/container_1492195730173_0001_01_000001/container_tokens
      DATAPROC_MASTER_COMPONENTS=hadoop-hdfs-namenode hadoop-yarn-resourcemanager mysql-server
      YARN_NICENESS=0
      HOME=/home/
      CONTAINER_ID=container_1492195730173_0001_01_000001
      MALLOC_ARENA_MAX=4
      2017-04-14 18:56:50,236 INFO com.datatorrent.stram.StreamingAppMaster: Initializing Application Master.
      2017-04-14 18:56:50,311 INFO com.datatorrent.stram.StreamingAppMasterService: Application master, appId=1, clustertimestamp=1492195730173, attemptId=1
      2017-04-14 18:56:52,099 INFO com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase: GHFS version: 1.6.0-hadoop2
      2017-04-14 18:56:52,491 INFO com.datatorrent.common.util.AsyncFSStorageAgent: using /hadoop/yarn/nm-local-dir/usercache/dhalperi/appcache/application_1492195730173_0001/container_1492195730173_0001_01_000001/tmp/chkp5833413517908828871 as the basepath for checkpointing.
      2017-04-14 18:56:56,041 INFO com.datatorrent.stram.FSRecoveryHandler: Creating hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001/recovery/log
      2017-04-14 18:56:56,172 INFO com.datatorrent.stram.StreamingAppMasterService: Starting application with 10 operators in 10 containers
      2017-04-14 18:56:56,182 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Upper bound of the thread pool size is 500
      2017-04-14 18:56:56,183 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
      2017-04-14 18:56:56,201 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at gaming-apex-dhalperi-m/10.240.169.55:8030
      2017-04-14 18:56:56,224 INFO com.datatorrent.stram.StreamingContainerParent: Config: Configuration: core-default.xml, core-site.xml, yarn-default.xml, yarn-site.xml, mapred-default.xml, mapred-site.xml, hdfs-default.xml, hdfs-site.xml
      2017-04-14 18:56:56,224 INFO com.datatorrent.stram.StreamingContainerParent: Listener thread count 30
      2017-04-14 18:56:56,230 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
      2017-04-14 18:56:56,235 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 57363
      2017-04-14 18:56:56,256 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
      2017-04-14 18:56:56,257 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 57363: starting
      2017-04-14 18:56:56,289 INFO com.datatorrent.stram.StreamingContainerParent: Container callback server listening at gaming-apex-dhalperi-w-15/10.240.169.52:57363
      2017-04-14 18:56:56,334 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
      2017-04-14 18:56:56,417 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
      2017-04-14 18:56:56,424 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.stram is not defined
      2017-04-14 18:56:56,432 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
      2017-04-14 18:56:56,434 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context stram
      2017-04-14 18:56:56,434 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
      2017-04-14 18:56:56,435 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
      2017-04-14 18:56:56,437 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /stram/*
      2017-04-14 18:56:56,437 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
      2017-04-14 18:56:56,658 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
      2017-04-14 18:56:56,659 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 32869
      2017-04-14 18:57:08,363 WARN com.datatorrent.stram.webapp.TypeGraphFactory: The size of precomputed type graph is 8136 KB
      2017-04-14 18:57:08,838 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app stram started at 32869
      2017-04-14 18:57:08,838 INFO com.datatorrent.stram.StreamingAppMasterService: Started web service at port: 32869
      2017-04-14 18:57:08,839 INFO com.datatorrent.stram.StreamingAppMasterService: Setting tracking URL to: gaming-apex-dhalperi-w-15:32869
      2017-04-14 18:57:08,849 INFO com.datatorrent.stram.StreamingAppMasterService: Starting ApplicationMaster
      2017-04-14 18:57:08,850 INFO com.datatorrent.stram.StreamingAppMasterService: number of tokens: 1
      2017-04-14 18:57:08,908 INFO com.datatorrent.stram.StreamingAppMasterService: Max mem 12288m, Min mem 1024m, Max vcores 32000 and Min vcores 1 capabililty of resources in this cluster 
      2017-04-14 18:57:08,909 INFO com.datatorrent.stram.StreamingAppMasterService: Blacklist removal time in millis = 3600000, max consecutive node failure count = 2147483647
      2017-04-14 18:57:08,913 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at gaming-apex-dhalperi-m/10.240.169.55:8032
      2017-04-14 18:57:10,127 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=9,name=SumTeamScores/GroupByKey]
      2017-04-14 18:57:10,127 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=1,name=TextIO.Read/Read]
      2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=4,name=FixedWindows/Window.Assign]
      2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]
      2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=2,name=ParseGameEvent/ParMultiDo(ParseEvent)]
      2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)]
      2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
      2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=5,name=SumTeamScores/ParDo(KeyScoreByTeam)/ParMultiDo(KeyScoreByTeam)]
      2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)]
      2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=3,name=SetTimestamps/ParMultiDo(SetTimestamps)]
      2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 18:57:10,132 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[0], Capability[<memory:1536, vCores:1>]Priority[1], Capability[<memory:1536, vCores:1>]Priority[2], Capability[<memory:1024, vCores:1>]Priority[3], Capability[<memory:1536, vCores:1>]Priority[4], Capability[<memory:1536, vCores:1>]Priority[5], Capability[<memory:1536, vCores:1>]Priority[6], Capability[<memory:1536, vCores:1>]Priority[7], Capability[<memory:1536, vCores:1>]Priority[8], Capability[<memory:1536, vCores:1>]Priority[9]]
      2017-04-14 18:57:10,132 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[0] on host: [null]
      2017-04-14 18:57:10,137 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[1] on host: [null]
      2017-04-14 18:57:10,137 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[2] on host: [null]
      2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1024, vCores:1>]Priority[3] on host: [null]
      2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[4] on host: [null]
      2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[5] on host: [null]
      2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[6] on host: [null]
      2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[7] on host: [null]
      2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[8] on host: [null]
      2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[9] on host: [null]
      2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:37663
      2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:50931
      2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:33276
      2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:37140
      2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928
      2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:55762
      2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743
      2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:46370
      2017-04-14 18:57:11,217 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054
      2017-04-14 18:57:11,217 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:39804
      2017-04-14 18:57:11,218 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000002, containerNode=gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:37663, containerNodeURI=gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority0
      2017-04-14 18:57:11,245 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000002
      2017-04-14 18:57:11,251 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 18:57:11,307 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 18:57:11,307 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000002
      2017-04-14 18:57:11,307 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:37663 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000002 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 18:57:11,317 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000003, containerNode=gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:50931, containerNodeURI=gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority1
      2017-04-14 18:57:11,321 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000002
      2017-04-14 18:57:11,325 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000003
      2017-04-14 18:57:11,326 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 18:57:11,332 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 18:57:11,332 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000003
      2017-04-14 18:57:11,334 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:50931 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000003 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 18:57:11,334 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:37663
      2017-04-14 18:57:11,339 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000004, containerNode=gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:33276, containerNodeURI=gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority2
      2017-04-14 18:57:11,342 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000003
      2017-04-14 18:57:11,345 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000004
      2017-04-14 18:57:11,346 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 18:57:11,350 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 18:57:11,351 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000004
      2017-04-14 18:57:11,351 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:33276 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000004 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 18:57:11,352 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000005, containerNode=gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:37140, containerNodeURI=gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:8042, containerResourceMemory1024, priority3
      2017-04-14 18:57:11,359 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000005
      2017-04-14 18:57:11,360 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 18:57:11,361 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000004
      2017-04-14 18:57:11,371 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 18:57:11,372 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx805306368  for container container_1492195730173_0001_01_000005
      2017-04-14 18:57:11,372 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:37140 command: $JAVA_HOME/bin/java  -Xmx805306368  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000005 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 18:57:11,373 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000006, containerNode=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928, containerNodeURI=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority4
      2017-04-14 18:57:11,373 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000005
      2017-04-14 18:57:11,384 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000006
      2017-04-14 18:57:11,384 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 18:57:11,387 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 18:57:11,388 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000006
      2017-04-14 18:57:11,388 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000006 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 18:57:11,389 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000007, containerNode=gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:55762, containerNodeURI=gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority5
      2017-04-14 18:57:11,394 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000007
      2017-04-14 18:57:11,394 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 18:57:11,401 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000006
      2017-04-14 18:57:11,401 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 18:57:11,406 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000007
      2017-04-14 18:57:11,407 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:55762 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000007 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 18:57:11,407 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000008, containerNode=gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743, containerNodeURI=gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority6
      2017-04-14 18:57:11,412 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:37140
      2017-04-14 18:57:11,413 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000007
      2017-04-14 18:57:11,415 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000008
      2017-04-14 18:57:11,415 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 18:57:11,418 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:33276
      2017-04-14 18:57:11,421 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 18:57:11,422 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000008
      2017-04-14 18:57:11,422 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000008 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 18:57:11,422 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000009, containerNode=gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:46370, containerNodeURI=gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority7
      2017-04-14 18:57:11,426 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000008
      2017-04-14 18:57:11,430 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000009
      2017-04-14 18:57:11,430 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 18:57:11,433 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:50931
      2017-04-14 18:57:11,436 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 18:57:11,436 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000009
      2017-04-14 18:57:11,436 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:46370 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000009 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 18:57:11,437 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000010, containerNode=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054, containerNodeURI=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority8
      2017-04-14 18:57:11,437 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000009
      2017-04-14 18:57:11,449 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743
      2017-04-14 18:57:11,449 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000010
      2017-04-14 18:57:11,450 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 18:57:11,454 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:55762
      2017-04-14 18:57:11,454 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 18:57:11,455 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000010
      2017-04-14 18:57:11,455 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000010 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 18:57:11,455 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000011, containerNode=gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:39804, containerNodeURI=gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority9
      2017-04-14 18:57:11,459 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928
      2017-04-14 18:57:11,463 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000011
      2017-04-14 18:57:11,463 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 18:57:11,460 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000010
      2017-04-14 18:57:11,473 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:46370
      2017-04-14 18:57:11,474 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054
      2017-04-14 18:57:11,482 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 18:57:11,482 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000011
      2017-04-14 18:57:11,482 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:39804 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000011 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 18:57:11,490 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000011
      2017-04-14 18:57:11,500 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:39804
      2017-04-14 18:57:13,756 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000010] Entering heartbeat loop.. context: PTContainer[id=7(container_1492195730173_0001_01_000010),state=ALLOCATED,operators=[PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)]]]
      2017-04-14 18:57:14,770 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000010 buffer server: gaming-apex-dhalperi-w-15:55080
      2017-04-14 18:57:14,941 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000008] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000008),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 18:57:15,054 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000002] Entering heartbeat loop.. context: PTContainer[id=9(container_1492195730173_0001_01_000002),state=ALLOCATED,operators=[PTOperator[id=9,name=SumTeamScores/GroupByKey]]]
      2017-04-14 18:57:15,073 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000009] Entering heartbeat loop.. context: PTContainer[id=5(container_1492195730173_0001_01_000009),state=ALLOCATED,operators=[PTOperator[id=5,name=SumTeamScores/ParDo(KeyScoreByTeam)/ParMultiDo(KeyScoreByTeam)]]]
      2017-04-14 18:57:15,106 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000003] Entering heartbeat loop.. context: PTContainer[id=1(container_1492195730173_0001_01_000003),state=ALLOCATED,operators=[PTOperator[id=1,name=TextIO.Read/Read]]]
      2017-04-14 18:57:15,125 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000011] Entering heartbeat loop.. context: PTContainer[id=3(container_1492195730173_0001_01_000011),state=ALLOCATED,operators=[PTOperator[id=3,name=SetTimestamps/ParMultiDo(SetTimestamps)]]]
      2017-04-14 18:57:15,232 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000006] Entering heartbeat loop.. context: PTContainer[id=2(container_1492195730173_0001_01_000006),state=ALLOCATED,operators=[PTOperator[id=2,name=ParseGameEvent/ParMultiDo(ParseEvent)]]]
      2017-04-14 18:57:15,333 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000007] Entering heartbeat loop.. context: PTContainer[id=8(container_1492195730173_0001_01_000007),state=ALLOCATED,operators=[PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)]]]
      2017-04-14 18:57:15,346 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000004] Entering heartbeat loop.. context: PTContainer[id=4(container_1492195730173_0001_01_000004),state=ALLOCATED,operators=[PTOperator[id=4,name=FixedWindows/Window.Assign]]]
      2017-04-14 18:57:15,472 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000005] Entering heartbeat loop.. context: PTContainer[id=10(container_1492195730173_0001_01_000005),state=ALLOCATED,operators=[PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]]
      2017-04-14 18:57:15,954 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000008 buffer server: gaming-apex-dhalperi-w-3:60148
      2017-04-14 18:57:16,066 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000002 buffer server: gaming-apex-dhalperi-w-9:52966
      2017-04-14 18:57:16,085 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000009 buffer server: gaming-apex-dhalperi-w-2:41280
      2017-04-14 18:57:16,116 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000003 buffer server: gaming-apex-dhalperi-w-11:41089
      2017-04-14 18:57:16,136 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000011 buffer server: gaming-apex-dhalperi-w-18:50437
      2017-04-14 18:57:16,244 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000006 buffer server: gaming-apex-dhalperi-w-16:55760
      2017-04-14 18:57:16,345 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000007 buffer server: gaming-apex-dhalperi-w-12:42777
      2017-04-14 18:57:16,359 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000004 buffer server: gaming-apex-dhalperi-w-5:58095
      2017-04-14 18:57:16,485 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000005 buffer server: gaming-apex-dhalperi-w-1:55004
      2017-04-14 19:00:14,948 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196414947, last window id change time 1492196354898, window processing timeout millis 60000
      2017-04-14 19:00:14,948 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=9,name=SumTeamScores/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196414947, last window id change time 1492196354896, window processing timeout millis 60000
      2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196355249, window processing timeout millis 60000
      2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196355318, window processing timeout millis 60000
      2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196354898, window processing timeout millis 60000
      2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=9,name=SumTeamScores/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196354896, window processing timeout millis 60000
      2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196354966, window processing timeout millis 60000
      2017-04-14 19:00:15,951 INFO com.datatorrent.stram.StreamingContainerManager: Blocked operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] container PTContainer[id=6(container_1492195730173_0001_01_000008),state=ACTIVE] time 60701ms
      2017-04-14 19:00:16,951 INFO com.datatorrent.stram.StreamingAppMasterService: Requested stop container container_1492195730173_0001_01_000008
      2017-04-14 19:00:16,951 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: STOP_CONTAINER for Container container_1492195730173_0001_01_000008
      2017-04-14 19:00:16,953 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196355249, window processing timeout millis 60000
      2017-04-14 19:00:16,953 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196355318, window processing timeout millis 60000
      2017-04-14 19:00:16,954 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196354898, window processing timeout millis 60000
      2017-04-14 19:00:16,954 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=9,name=SumTeamScores/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196354896, window processing timeout millis 60000
      2017-04-14 19:00:16,954 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196354966, window processing timeout millis 60000
      2017-04-14 19:00:16,954 INFO com.datatorrent.stram.StreamingContainerManager: Blocked operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] container PTContainer[id=6(container_1492195730173_0001_01_000008),state=ACTIVE] time 61704ms
      2017-04-14 19:00:16,957 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743
      2017-04-14 19:00:17,962 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000008, state=COMPLETE, exitStatus=-105, diagnostics=Container killed by the ApplicationMaster.
      Container killed on request. Exit code is 143
      Container exited with a non-zero exit code 143
      
      2017-04-14 19:00:17,962 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000008@gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743
      2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196355249, window processing timeout millis 60000
      2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196355318, window processing timeout millis 60000
      2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196354898, window processing timeout millis 60000
      2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=9,name=SumTeamScores/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196354896, window processing timeout millis 60000
      2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196354966, window processing timeout millis 60000
      2017-04-14 19:00:17,965 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
      2017-04-14 19:00:19,013 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
      2017-04-14 19:00:19,013 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 19:00:19,014 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[10]]
      2017-04-14 19:00:19,014 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[10] on host: [null]
      2017-04-14 19:00:20,021 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000012, containerNode=gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743, containerNodeURI=gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority10
      2017-04-14 19:00:20,022 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000008
      2017-04-14 19:00:20,024 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000012
      2017-04-14 19:00:20,024 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 19:00:20,034 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 19:00:20,035 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000012
      2017-04-14 19:00:20,035 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 19:00:20,035 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000012
      2017-04-14 19:00:20,036 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743
      2017-04-14 19:00:21,762 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000012] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000012),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:00:22,774 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000012 buffer server: gaming-apex-dhalperi-w-3:54029
      2017-04-14 19:00:23,006 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-3]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
      Serialization trace:
      activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
      	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
      	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
      	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
      	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
      	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
      	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
      	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
      	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
       context: PTContainer[id=6(container_1492195730173_0001_01_000012),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:00:23,011 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-3]]]]
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
      	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
      	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
      Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
      Serialization trace:
      activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
      	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
      	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
      	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
      	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
      	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
      	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
      	... 2 more
       context: PTContainer[id=6(container_1492195730173_0001_01_000012),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:00:24,048 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000012, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
      Container id: container_1492195730173_0001_01_000012
      Exit code: 1
      Stack trace: ExitCodeException exitCode=1: 
      	at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
      	at org.apache.hadoop.util.Shell.run(Shell.java:479)
      	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
      	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      
      Container exited with a non-zero exit code 1
      
      2017-04-14 19:00:24,049 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000012@gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743
      2017-04-14 19:00:24,049 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
      2017-04-14 19:00:25,072 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
      2017-04-14 19:00:25,072 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 19:00:25,072 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[11]]
      2017-04-14 19:00:25,072 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[11] on host: [null]
      2017-04-14 19:00:26,078 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000013, containerNode=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928, containerNodeURI=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority11
      2017-04-14 19:00:26,079 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000012
      2017-04-14 19:00:26,081 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000013
      2017-04-14 19:00:26,081 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 19:00:26,084 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 19:00:26,084 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000013
      2017-04-14 19:00:26,085 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000013 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 19:00:26,085 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000013
      2017-04-14 19:00:26,096 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928
      2017-04-14 19:00:30,185 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000013] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000013),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:00:31,201 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000013 buffer server: gaming-apex-dhalperi-w-16:43881
      2017-04-14 19:00:31,590 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-16]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
      Serialization trace:
      activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
      	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
      	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
      	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
      	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
      	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
      	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
      	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
      	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
       context: PTContainer[id=6(container_1492195730173_0001_01_000013),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:00:31,618 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-16]]]]
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
      	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
      	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
      Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
      Serialization trace:
      activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
      	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
      	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
      	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
      	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
      	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
      	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
      	... 2 more
       context: PTContainer[id=6(container_1492195730173_0001_01_000013),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:00:34,104 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000013, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
      Container id: container_1492195730173_0001_01_000013
      Exit code: 1
      Stack trace: ExitCodeException exitCode=1: 
      	at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
      	at org.apache.hadoop.util.Shell.run(Shell.java:479)
      	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
      	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      
      Container exited with a non-zero exit code 1
      
      2017-04-14 19:00:34,105 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000013@gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928
      2017-04-14 19:00:34,105 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
      2017-04-14 19:00:35,127 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
      2017-04-14 19:00:35,127 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 19:00:35,127 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[12]]
      2017-04-14 19:00:35,127 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[12] on host: [null]
      2017-04-14 19:00:36,135 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000014, containerNode=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054, containerNodeURI=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority12
      2017-04-14 19:00:36,135 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000013
      2017-04-14 19:00:36,138 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000014
      2017-04-14 19:00:36,138 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 19:00:36,144 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 19:00:36,144 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000014
      2017-04-14 19:00:36,144 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000014 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 19:00:36,144 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000014
      2017-04-14 19:00:36,145 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054
      2017-04-14 19:00:37,902 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000014] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000014),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:00:38,913 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000014 buffer server: gaming-apex-dhalperi-w-15:59584
      2017-04-14 19:00:39,107 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-15]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
      Serialization trace:
      activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
      	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
      	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
      	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
      	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
      	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
      	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
      	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
      	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
       context: PTContainer[id=6(container_1492195730173_0001_01_000014),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:00:39,112 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-15]]]]
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
      	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
      	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
      Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
      Serialization trace:
      activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
      	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
      	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
      	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
      	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
      	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
      	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
      	... 2 more
       context: PTContainer[id=6(container_1492195730173_0001_01_000014),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:00:40,160 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000014, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
      Container id: container_1492195730173_0001_01_000014
      Exit code: 1
      Stack trace: ExitCodeException exitCode=1: 
      	at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
      	at org.apache.hadoop.util.Shell.run(Shell.java:479)
      	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
      	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      
      Container exited with a non-zero exit code 1
      
      2017-04-14 19:00:40,161 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000014@gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054
      2017-04-14 19:00:40,161 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
      2017-04-14 19:00:41,199 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
      2017-04-14 19:00:41,199 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 19:00:41,199 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[13]]
      2017-04-14 19:00:41,199 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[13] on host: [null]
      2017-04-14 19:00:42,206 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733
      2017-04-14 19:00:42,206 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000015, containerNode=gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733, containerNodeURI=gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority13
      2017-04-14 19:00:42,206 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000014
      2017-04-14 19:00:42,208 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000015
      2017-04-14 19:00:42,208 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 19:00:42,212 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 19:00:42,212 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000015
      2017-04-14 19:00:42,212 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000015 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 19:00:42,212 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Set NMClientAsync thread pool size to 21 as the number of nodes to talk to is 11
      2017-04-14 19:00:42,214 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000015
      2017-04-14 19:00:42,219 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733
      2017-04-14 19:00:45,476 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000015] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000015),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:00:46,487 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000015 buffer server: gaming-apex-dhalperi-w-8:57550
      2017-04-14 19:00:46,661 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-8]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
      Serialization trace:
      activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
      	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
      	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
      	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
      	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
      	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
      	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
      	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
      	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
       context: PTContainer[id=6(container_1492195730173_0001_01_000015),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:00:46,668 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-8]]]]
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
      	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
      	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
      Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
      Serialization trace:
      activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
      	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
      	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
      	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
      	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
      	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
      	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
      	... 2 more
       context: PTContainer[id=6(container_1492195730173_0001_01_000015),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:00:47,225 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000015, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
      Container id: container_1492195730173_0001_01_000015
      Exit code: 1
      Stack trace: ExitCodeException exitCode=1: 
      	at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
      	at org.apache.hadoop.util.Shell.run(Shell.java:479)
      	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
      	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      
      Container exited with a non-zero exit code 1
      
      2017-04-14 19:00:47,226 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000015@gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733
      2017-04-14 19:00:47,226 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
      2017-04-14 19:00:48,263 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
      2017-04-14 19:00:48,263 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 19:00:48,264 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[14]]
      2017-04-14 19:00:48,264 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[14] on host: [null]
      2017-04-14 19:00:49,271 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000016, containerNode=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928, containerNodeURI=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority14
      2017-04-14 19:00:49,272 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000015
      2017-04-14 19:00:49,274 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000016
      2017-04-14 19:00:49,274 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 19:00:49,277 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 19:00:49,277 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000016
      2017-04-14 19:00:49,277 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000016 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 19:00:49,278 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000016
      2017-04-14 19:00:49,279 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928
      2017-04-14 19:00:52,751 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000016] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000016),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:00:53,775 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000016 buffer server: gaming-apex-dhalperi-w-16:52403
      2017-04-14 19:00:54,135 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-16]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
      Serialization trace:
      activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
      	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
      	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
      	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
      	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
      	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
      	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
      	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
      	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
       context: PTContainer[id=6(container_1492195730173_0001_01_000016),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:00:54,145 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-16]]]]
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
      	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
      	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
      Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
      Serialization trace:
      activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
      	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
      	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
      	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
      	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
      	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
      	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
      	... 2 more
       context: PTContainer[id=6(container_1492195730173_0001_01_000016),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:00:55,291 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000016, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
      Container id: container_1492195730173_0001_01_000016
      Exit code: 1
      Stack trace: ExitCodeException exitCode=1: 
      	at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
      	at org.apache.hadoop.util.Shell.run(Shell.java:479)
      	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
      	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      
      Container exited with a non-zero exit code 1
      
      2017-04-14 19:00:55,292 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000016@gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928
      2017-04-14 19:00:55,292 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
      2017-04-14 19:00:56,328 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
      2017-04-14 19:00:56,328 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 19:00:56,329 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[15]]
      2017-04-14 19:00:56,329 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[15] on host: [null]
      2017-04-14 19:00:57,337 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866
      2017-04-14 19:00:57,337 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000017, containerNode=gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866, containerNodeURI=gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority15
      2017-04-14 19:00:57,337 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000016
      2017-04-14 19:00:57,339 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000017
      2017-04-14 19:00:57,340 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 19:00:57,348 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 19:00:57,348 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000017
      2017-04-14 19:00:57,348 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000017 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 19:00:57,349 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000017
      2017-04-14 19:00:57,354 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866
      2017-04-14 19:01:01,038 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000017] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000017),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:01:02,049 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000017 buffer server: gaming-apex-dhalperi-w-14:50842
      2017-04-14 19:01:02,198 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-14]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
      Serialization trace:
      activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
      	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
      	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
      	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
      	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
      	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
      	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
      	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
      	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
       context: PTContainer[id=6(container_1492195730173_0001_01_000017),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:01:02,204 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-14]]]]
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
      	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
      	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
      Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
      Serialization trace:
      activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
      	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
      	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
      	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
      	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
      	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
      	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
      	... 2 more
       context: PTContainer[id=6(container_1492195730173_0001_01_000017),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:01:03,363 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000017, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
      Container id: container_1492195730173_0001_01_000017
      Exit code: 1
      Stack trace: ExitCodeException exitCode=1: 
      	at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
      	at org.apache.hadoop.util.Shell.run(Shell.java:479)
      	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
      	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      
      Container exited with a non-zero exit code 1
      
      2017-04-14 19:01:03,363 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000017@gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866
      2017-04-14 19:01:03,364 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
      2017-04-14 19:01:04,387 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
      2017-04-14 19:01:04,387 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
      2017-04-14 19:01:04,388 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[16]]
      2017-04-14 19:01:04,388 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[16] on host: [null]
      2017-04-14 19:01:05,396 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:43371
      2017-04-14 19:01:05,396 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000018, containerNode=gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:43371, containerNodeURI=gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority16
      2017-04-14 19:01:05,397 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000017
      2017-04-14 19:01:05,399 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000018
      2017-04-14 19:01:05,399 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
      2017-04-14 19:01:05,401 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
      2017-04-14 19:01:05,401 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000018
      2017-04-14 19:01:05,401 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:43371 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000018 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
      2017-04-14 19:01:05,402 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000018
      2017-04-14 19:01:05,407 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:43371
      2017-04-14 19:01:09,197 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000018] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000018),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:01:10,209 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000018 buffer server: gaming-apex-dhalperi-w-6:49764
      2017-04-14 19:01:10,377 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-6]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
      Serialization trace:
      activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
      	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
      	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
      	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
      	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
      	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
      	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
      	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
      	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
       context: PTContainer[id=6(container_1492195730173_0001_01_000018),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:01:10,385 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-6]]]]
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
      	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
      	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
      Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
      Serialization trace:
      activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
      	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
      	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
      	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
      	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
      	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
      	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
      	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
      	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
      	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
      	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
      	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
      	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
      	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
      	... 2 more
       context: PTContainer[id=6(container_1492195730173_0001_01_000018),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
      2017-04-14 19:01:11,416 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000018, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
      Container id: container_1492195730173_0001_01_000018
      Exit code: 1
      

      Attachments

        Issue Links

          Activity

            People

              thw Thomas Weise
              dhalperi Dan Halperin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: