[BEAM-1981] Serialization error with TimerInternals in ApexGroupByKeyOperator

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: runner-apex
• Assignee: Thomas Weise
• Reporter: Dan Halperin
• Created: 14/Apr/17 21:47
• Updated: 16/May/20 13:56
• Resolved: 28/Apr/17 19:37

Description

Logs below. We tried switching to Java serialization, but that didn't work. We made the field transient (which is broken but let us make progress) and that did.

Stack trace

com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
Serialization trace:
activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
 context: PTContainer[id=6(container_1492195730173_0001_01_000012),state=ACTIVE,operators=[

Larger logs with more scope:

	
2017-04-14 18:56:49,961 INFO com.datatorrent.stram.StreamingAppMaster: Master starting with classpath: ./portability-demo-bundled-apex.jar:/etc/hadoop/conf:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-auth-2.7.3.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-common-2.7.3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7.3.jar:/usr/lib/hadoop/hadoop-nfs-2.7.3.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/hadoop-common-2.7.3.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/snappy-java-1.0.5.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/hadoop-lzo-0.4.19.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/bigquery-connector-0.10.1-hadoop2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/avro-1.7.7.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/gcs-connector-1.6.0-hadoop2.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/zookeeper-3.4.6.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jetty-6.1.26.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/httpclient-4.2.5.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/httpcore-4.2.5.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.5.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/hadoop-auth-2.7.3.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-1.7.4.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7.3.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/hadoop-extras-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-aws-2.7.3.jar:/usr/lib/hadoop-mapreduce/avro-1.7.7.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar:/usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-archives.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/zookeeper-3.4.6.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26.jar:/usr/lib/hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/joda-time-2.9.9.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.42.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix-2.7.3.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/usr/lib/hadoop-mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.7.3.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/httpclient-4.2.5.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar:/usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/jackson-annotations-2.2.3.jar:/usr/lib/hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/jackson-databind-2.2.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.7.3.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/httpcore-4.2.5.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3.jar:/usr/lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop-mapreduce/hadoop-aws.jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/jackson-core-2.2.3.jar:/usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/usr/lib/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.5.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.7.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/lib/*'
2017-04-14 18:56:49,966 INFO com.datatorrent.stram.StreamingAppMaster: version: 3.5.0 from rev: 6de8828 branch: 6de8828e4f3d5734d0a6f9c1be0aa7057cb60ac8 by Thomas Weise on 06.12.2016 @ 22:11:51 PST
2017-04-14 18:56:49,967 INFO com.datatorrent.stram.StreamingAppMaster: appmaster env:
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
HADOOP_CONF_DIR=/etc/hadoop/conf
MAX_APP_ATTEMPTS=2
DATAPROC_MASTER_STANDALONE_COMPONENTS=hadoop-hdfs-secondarynamenode
DATAPROC_COMMON_COMPONENTS=openjdk-8-jdk libjansi-java python-numpy libmysql-java hadoop-client hive pig spark-core spark-python spark-r autofs nfs-common libhdfs0 libsnappy1 libatlas3-base libopenblas-base libapr1 vim git bash-completion spark-yarn-shuffle spark-datanucleus spark-extras hadoop-lzo
MAIL=/var/mail/yarn
LD_LIBRARY_PATH=:/usr/lib/hadoop/lib/native:/usr/lib/hadoop/lib/native
HADOOP_HOME_WARN_SUPPRESS=true
DATAPROC_WORKER_COMPONENTS=hadoop-hdfs-datanode hadoop-yarn-nodemanager
LOGNAME=dhalperi
JVM_PID=2123
JSVC_HOME=/usr/lib/bigtop-utils
PWD=/hadoop/yarn/nm-local-dir/usercache/dhalperi/appcache/application_1492195730173_0001/container_1492195730173_0001_01_000001
HADOOP_YARN_USER=yarn
HADOOP_PREFIX=/usr/lib/hadoop
LOCAL_DIRS=/hadoop/yarn/nm-local-dir/usercache/dhalperi/appcache/application_1492195730173_0001
YARN_IDENT_STRING=yarn
SHELL=/bin/bash
ALPN_JAR=/usr/local/share/google/alpn/alpn-boot-8.1.7.v20160121.jar
YARN_CONF_DIR=/etc/hadoop/conf
_SYSTEMCTL_SKIP_REDIRECT=true
LOG_DIRS=/var/log/hadoop-yarn/userlogs/application_1492195730173_0001/container_1492195730173_0001_01_000001
NM_AUX_SERVICE_mapreduce_shuffle=AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=

BDUTIL_DIR=/usr/local/share/google/dataproc/bdutil-dataproc-20170410-124434-RC1
HADOOP_YARN_HOME=/usr/lib/hadoop-yarn
YARN_PID_DIR=/var/run/hadoop-yarn
DATAPROC_MASTER_EXCLUSIVE_COMPONENTS=hadoop-mapreduce-historyserver hive-metastore hive-server2 nfs-kernel-server spark-history-server
NM_AUX_SERVICE_spark_shuffle=
HADOOP_HOME=/usr/lib/hadoop
SHLVL=5
YARN_ROOT_LOGGER=INFO,RFA
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
LANG=en_US.UTF-8
APP_SUBMIT_TIME_ENV=1492196207488
NM_HOST=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal
YARN_LOGFILE=yarn-yarn-nodemanager-gaming-apex-dhalperi-w-15.log
HADOOP_USER_NAME=dhalperi
DATAPROC_AGENT_JAR=/usr/local/share/google/dataproc/agent-20170410-124434-RC1.jar
HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec
YARN_LOG_DIR=/var/log/hadoop-yarn
HADOOP_HDFS_HOME=/usr/lib/hadoop-hdfs
HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce
HADOOP_COMMON_HOME=/usr/lib/hadoop
_=/usr/lib/jvm/java-8-openjdk-amd64/bin/java
APPLICATION_WEB_PROXY_BASE=/proxy/application_1492195730173_0001
NM_HTTP_PORT=8042
NM_PORT=53054
USER=dhalperi
CLASSPATH=./*:/etc/hadoop/conf:/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-yarn/*:/usr/lib/hadoop-yarn/lib/*'
DATAPROC_MASTER_HA_COMPONENTS=hadoop-hdfs-journalnode hadoop-hdfs-zkfc zookeeper-server
HADOOP_TOKEN_FILE_LOCATION=/hadoop/yarn/nm-local-dir/usercache/dhalperi/appcache/application_1492195730173_0001/container_1492195730173_0001_01_000001/container_tokens
DATAPROC_MASTER_COMPONENTS=hadoop-hdfs-namenode hadoop-yarn-resourcemanager mysql-server
YARN_NICENESS=0
HOME=/home/
CONTAINER_ID=container_1492195730173_0001_01_000001
MALLOC_ARENA_MAX=4
2017-04-14 18:56:50,236 INFO com.datatorrent.stram.StreamingAppMaster: Initializing Application Master.
2017-04-14 18:56:50,311 INFO com.datatorrent.stram.StreamingAppMasterService: Application master, appId=1, clustertimestamp=1492195730173, attemptId=1
2017-04-14 18:56:52,099 INFO com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase: GHFS version: 1.6.0-hadoop2
2017-04-14 18:56:52,491 INFO com.datatorrent.common.util.AsyncFSStorageAgent: using /hadoop/yarn/nm-local-dir/usercache/dhalperi/appcache/application_1492195730173_0001/container_1492195730173_0001_01_000001/tmp/chkp5833413517908828871 as the basepath for checkpointing.
2017-04-14 18:56:56,041 INFO com.datatorrent.stram.FSRecoveryHandler: Creating hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001/recovery/log
2017-04-14 18:56:56,172 INFO com.datatorrent.stram.StreamingAppMasterService: Starting application with 10 operators in 10 containers
2017-04-14 18:56:56,182 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Upper bound of the thread pool size is 500
2017-04-14 18:56:56,183 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
2017-04-14 18:56:56,201 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at gaming-apex-dhalperi-m/10.240.169.55:8030
2017-04-14 18:56:56,224 INFO com.datatorrent.stram.StreamingContainerParent: Config: Configuration: core-default.xml, core-site.xml, yarn-default.xml, yarn-site.xml, mapred-default.xml, mapred-site.xml, hdfs-default.xml, hdfs-site.xml
2017-04-14 18:56:56,224 INFO com.datatorrent.stram.StreamingContainerParent: Listener thread count 30
2017-04-14 18:56:56,230 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2017-04-14 18:56:56,235 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 57363
2017-04-14 18:56:56,256 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2017-04-14 18:56:56,257 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 57363: starting
2017-04-14 18:56:56,289 INFO com.datatorrent.stram.StreamingContainerParent: Container callback server listening at gaming-apex-dhalperi-w-15/10.240.169.52:57363
2017-04-14 18:56:56,334 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2017-04-14 18:56:56,417 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2017-04-14 18:56:56,424 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.stram is not defined
2017-04-14 18:56:56,432 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2017-04-14 18:56:56,434 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context stram
2017-04-14 18:56:56,434 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2017-04-14 18:56:56,435 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2017-04-14 18:56:56,437 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /stram/*
2017-04-14 18:56:56,437 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2017-04-14 18:56:56,658 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2017-04-14 18:56:56,659 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 32869
2017-04-14 18:57:08,363 WARN com.datatorrent.stram.webapp.TypeGraphFactory: The size of precomputed type graph is 8136 KB
2017-04-14 18:57:08,838 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app stram started at 32869
2017-04-14 18:57:08,838 INFO com.datatorrent.stram.StreamingAppMasterService: Started web service at port: 32869
2017-04-14 18:57:08,839 INFO com.datatorrent.stram.StreamingAppMasterService: Setting tracking URL to: gaming-apex-dhalperi-w-15:32869
2017-04-14 18:57:08,849 INFO com.datatorrent.stram.StreamingAppMasterService: Starting ApplicationMaster
2017-04-14 18:57:08,850 INFO com.datatorrent.stram.StreamingAppMasterService: number of tokens: 1
2017-04-14 18:57:08,908 INFO com.datatorrent.stram.StreamingAppMasterService: Max mem 12288m, Min mem 1024m, Max vcores 32000 and Min vcores 1 capabililty of resources in this cluster 
2017-04-14 18:57:08,909 INFO com.datatorrent.stram.StreamingAppMasterService: Blacklist removal time in millis = 3600000, max consecutive node failure count = 2147483647
2017-04-14 18:57:08,913 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at gaming-apex-dhalperi-m/10.240.169.55:8032
2017-04-14 18:57:10,127 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=9,name=SumTeamScores/GroupByKey]
2017-04-14 18:57:10,127 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=1,name=TextIO.Read/Read]
2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=4,name=FixedWindows/Window.Assign]
2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]
2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=2,name=ParseGameEvent/ParMultiDo(ParseEvent)]
2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)]
2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=5,name=SumTeamScores/ParDo(KeyScoreByTeam)/ParMultiDo(KeyScoreByTeam)]
2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)]
2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=3,name=SetTimestamps/ParMultiDo(SetTimestamps)]
2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 18:57:10,132 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[0], Capability[<memory:1536, vCores:1>]Priority[1], Capability[<memory:1536, vCores:1>]Priority[2], Capability[<memory:1024, vCores:1>]Priority[3], Capability[<memory:1536, vCores:1>]Priority[4], Capability[<memory:1536, vCores:1>]Priority[5], Capability[<memory:1536, vCores:1>]Priority[6], Capability[<memory:1536, vCores:1>]Priority[7], Capability[<memory:1536, vCores:1>]Priority[8], Capability[<memory:1536, vCores:1>]Priority[9]]
2017-04-14 18:57:10,132 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[0] on host: [null]
2017-04-14 18:57:10,137 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[1] on host: [null]
2017-04-14 18:57:10,137 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[2] on host: [null]
2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1024, vCores:1>]Priority[3] on host: [null]
2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[4] on host: [null]
2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[5] on host: [null]
2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[6] on host: [null]
2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[7] on host: [null]
2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[8] on host: [null]
2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[9] on host: [null]
2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:37663
2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:50931
2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:33276
2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:37140
2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928
2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:55762
2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743
2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:46370
2017-04-14 18:57:11,217 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054
2017-04-14 18:57:11,217 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:39804
2017-04-14 18:57:11,218 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000002, containerNode=gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:37663, containerNodeURI=gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority0
2017-04-14 18:57:11,245 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000002
2017-04-14 18:57:11,251 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 18:57:11,307 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 18:57:11,307 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000002
2017-04-14 18:57:11,307 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:37663 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000002 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 18:57:11,317 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000003, containerNode=gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:50931, containerNodeURI=gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority1
2017-04-14 18:57:11,321 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000002
2017-04-14 18:57:11,325 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000003
2017-04-14 18:57:11,326 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 18:57:11,332 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 18:57:11,332 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000003
2017-04-14 18:57:11,334 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:50931 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000003 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 18:57:11,334 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:37663
2017-04-14 18:57:11,339 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000004, containerNode=gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:33276, containerNodeURI=gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority2
2017-04-14 18:57:11,342 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000003
2017-04-14 18:57:11,345 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000004
2017-04-14 18:57:11,346 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 18:57:11,350 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 18:57:11,351 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000004
2017-04-14 18:57:11,351 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:33276 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000004 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 18:57:11,352 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000005, containerNode=gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:37140, containerNodeURI=gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:8042, containerResourceMemory1024, priority3
2017-04-14 18:57:11,359 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000005
2017-04-14 18:57:11,360 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 18:57:11,361 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000004
2017-04-14 18:57:11,371 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 18:57:11,372 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx805306368  for container container_1492195730173_0001_01_000005
2017-04-14 18:57:11,372 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:37140 command: $JAVA_HOME/bin/java  -Xmx805306368  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000005 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 18:57:11,373 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000006, containerNode=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928, containerNodeURI=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority4
2017-04-14 18:57:11,373 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000005
2017-04-14 18:57:11,384 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000006
2017-04-14 18:57:11,384 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 18:57:11,387 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 18:57:11,388 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000006
2017-04-14 18:57:11,388 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000006 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 18:57:11,389 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000007, containerNode=gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:55762, containerNodeURI=gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority5
2017-04-14 18:57:11,394 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000007
2017-04-14 18:57:11,394 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 18:57:11,401 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000006
2017-04-14 18:57:11,401 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 18:57:11,406 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000007
2017-04-14 18:57:11,407 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:55762 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000007 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 18:57:11,407 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000008, containerNode=gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743, containerNodeURI=gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority6
2017-04-14 18:57:11,412 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:37140
2017-04-14 18:57:11,413 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000007
2017-04-14 18:57:11,415 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000008
2017-04-14 18:57:11,415 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 18:57:11,418 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:33276
2017-04-14 18:57:11,421 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 18:57:11,422 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000008
2017-04-14 18:57:11,422 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000008 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 18:57:11,422 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000009, containerNode=gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:46370, containerNodeURI=gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority7
2017-04-14 18:57:11,426 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000008
2017-04-14 18:57:11,430 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000009
2017-04-14 18:57:11,430 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 18:57:11,433 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:50931
2017-04-14 18:57:11,436 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 18:57:11,436 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000009
2017-04-14 18:57:11,436 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:46370 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000009 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 18:57:11,437 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000010, containerNode=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054, containerNodeURI=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority8
2017-04-14 18:57:11,437 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000009
2017-04-14 18:57:11,449 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743
2017-04-14 18:57:11,449 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000010
2017-04-14 18:57:11,450 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 18:57:11,454 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:55762
2017-04-14 18:57:11,454 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 18:57:11,455 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000010
2017-04-14 18:57:11,455 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000010 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 18:57:11,455 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000011, containerNode=gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:39804, containerNodeURI=gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority9
2017-04-14 18:57:11,459 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928
2017-04-14 18:57:11,463 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000011
2017-04-14 18:57:11,463 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 18:57:11,460 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000010
2017-04-14 18:57:11,473 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:46370
2017-04-14 18:57:11,474 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054
2017-04-14 18:57:11,482 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 18:57:11,482 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000011
2017-04-14 18:57:11,482 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:39804 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000011 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 18:57:11,490 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000011
2017-04-14 18:57:11,500 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:39804
2017-04-14 18:57:13,756 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000010] Entering heartbeat loop.. context: PTContainer[id=7(container_1492195730173_0001_01_000010),state=ALLOCATED,operators=[PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)]]]
2017-04-14 18:57:14,770 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000010 buffer server: gaming-apex-dhalperi-w-15:55080
2017-04-14 18:57:14,941 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000008] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000008),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 18:57:15,054 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000002] Entering heartbeat loop.. context: PTContainer[id=9(container_1492195730173_0001_01_000002),state=ALLOCATED,operators=[PTOperator[id=9,name=SumTeamScores/GroupByKey]]]
2017-04-14 18:57:15,073 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000009] Entering heartbeat loop.. context: PTContainer[id=5(container_1492195730173_0001_01_000009),state=ALLOCATED,operators=[PTOperator[id=5,name=SumTeamScores/ParDo(KeyScoreByTeam)/ParMultiDo(KeyScoreByTeam)]]]
2017-04-14 18:57:15,106 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000003] Entering heartbeat loop.. context: PTContainer[id=1(container_1492195730173_0001_01_000003),state=ALLOCATED,operators=[PTOperator[id=1,name=TextIO.Read/Read]]]
2017-04-14 18:57:15,125 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000011] Entering heartbeat loop.. context: PTContainer[id=3(container_1492195730173_0001_01_000011),state=ALLOCATED,operators=[PTOperator[id=3,name=SetTimestamps/ParMultiDo(SetTimestamps)]]]
2017-04-14 18:57:15,232 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000006] Entering heartbeat loop.. context: PTContainer[id=2(container_1492195730173_0001_01_000006),state=ALLOCATED,operators=[PTOperator[id=2,name=ParseGameEvent/ParMultiDo(ParseEvent)]]]
2017-04-14 18:57:15,333 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000007] Entering heartbeat loop.. context: PTContainer[id=8(container_1492195730173_0001_01_000007),state=ALLOCATED,operators=[PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)]]]
2017-04-14 18:57:15,346 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000004] Entering heartbeat loop.. context: PTContainer[id=4(container_1492195730173_0001_01_000004),state=ALLOCATED,operators=[PTOperator[id=4,name=FixedWindows/Window.Assign]]]
2017-04-14 18:57:15,472 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000005] Entering heartbeat loop.. context: PTContainer[id=10(container_1492195730173_0001_01_000005),state=ALLOCATED,operators=[PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]]
2017-04-14 18:57:15,954 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000008 buffer server: gaming-apex-dhalperi-w-3:60148
2017-04-14 18:57:16,066 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000002 buffer server: gaming-apex-dhalperi-w-9:52966
2017-04-14 18:57:16,085 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000009 buffer server: gaming-apex-dhalperi-w-2:41280
2017-04-14 18:57:16,116 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000003 buffer server: gaming-apex-dhalperi-w-11:41089
2017-04-14 18:57:16,136 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000011 buffer server: gaming-apex-dhalperi-w-18:50437
2017-04-14 18:57:16,244 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000006 buffer server: gaming-apex-dhalperi-w-16:55760
2017-04-14 18:57:16,345 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000007 buffer server: gaming-apex-dhalperi-w-12:42777
2017-04-14 18:57:16,359 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000004 buffer server: gaming-apex-dhalperi-w-5:58095
2017-04-14 18:57:16,485 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000005 buffer server: gaming-apex-dhalperi-w-1:55004
2017-04-14 19:00:14,948 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196414947, last window id change time 1492196354898, window processing timeout millis 60000
2017-04-14 19:00:14,948 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=9,name=SumTeamScores/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196414947, last window id change time 1492196354896, window processing timeout millis 60000
2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196355249, window processing timeout millis 60000
2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196355318, window processing timeout millis 60000
2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196354898, window processing timeout millis 60000
2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=9,name=SumTeamScores/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196354896, window processing timeout millis 60000
2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196354966, window processing timeout millis 60000
2017-04-14 19:00:15,951 INFO com.datatorrent.stram.StreamingContainerManager: Blocked operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] container PTContainer[id=6(container_1492195730173_0001_01_000008),state=ACTIVE] time 60701ms
2017-04-14 19:00:16,951 INFO com.datatorrent.stram.StreamingAppMasterService: Requested stop container container_1492195730173_0001_01_000008
2017-04-14 19:00:16,951 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: STOP_CONTAINER for Container container_1492195730173_0001_01_000008
2017-04-14 19:00:16,953 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196355249, window processing timeout millis 60000
2017-04-14 19:00:16,953 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196355318, window processing timeout millis 60000
2017-04-14 19:00:16,954 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196354898, window processing timeout millis 60000
2017-04-14 19:00:16,954 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=9,name=SumTeamScores/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196354896, window processing timeout millis 60000
2017-04-14 19:00:16,954 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196354966, window processing timeout millis 60000
2017-04-14 19:00:16,954 INFO com.datatorrent.stram.StreamingContainerManager: Blocked operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] container PTContainer[id=6(container_1492195730173_0001_01_000008),state=ACTIVE] time 61704ms
2017-04-14 19:00:16,957 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743
2017-04-14 19:00:17,962 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000008, state=COMPLETE, exitStatus=-105, diagnostics=Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

2017-04-14 19:00:17,962 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for [e-mail]:44743
2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196355249, window processing timeout millis 60000
2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196355318, window processing timeout millis 60000
2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196354898, window processing timeout millis 60000
2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=9,name=SumTeamScores/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196354896, window processing timeout millis 60000
2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196354966, window processing timeout millis 60000
2017-04-14 19:00:17,965 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
2017-04-14 19:00:19,013 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
2017-04-14 19:00:19,013 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 19:00:19,014 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[10]]
2017-04-14 19:00:19,014 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[10] on host: [null]
2017-04-14 19:00:20,021 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000012, containerNode=gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743, containerNodeURI=gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority10
2017-04-14 19:00:20,022 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000008
2017-04-14 19:00:20,024 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000012
2017-04-14 19:00:20,024 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 19:00:20,034 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 19:00:20,035 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000012
2017-04-14 19:00:20,035 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 19:00:20,035 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000012
2017-04-14 19:00:20,036 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743
2017-04-14 19:00:21,762 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000012] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000012),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:00:22,774 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000012 buffer server: gaming-apex-dhalperi-w-3:54029
2017-04-14 19:00:23,006 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-3]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
Serialization trace:
activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
 context: PTContainer[id=6(container_1492195730173_0001_01_000012),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:00:23,011 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-3]]]]
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
Serialization trace:
activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
	... 2 more
 context: PTContainer[id=6(container_1492195730173_0001_01_000012),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:00:24,048 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000012, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
Container id: container_1492195730173_0001_01_000012
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
	at org.apache.hadoop.util.Shell.run(Shell.java:479)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

2017-04-14 19:00:24,049 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for [e-mail]:44743
2017-04-14 19:00:24,049 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
2017-04-14 19:00:25,072 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
2017-04-14 19:00:25,072 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 19:00:25,072 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[11]]
2017-04-14 19:00:25,072 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[11] on host: [null]
2017-04-14 19:00:26,078 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000013, containerNode=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928, containerNodeURI=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority11
2017-04-14 19:00:26,079 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000012
2017-04-14 19:00:26,081 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000013
2017-04-14 19:00:26,081 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 19:00:26,084 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 19:00:26,084 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000013
2017-04-14 19:00:26,085 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000013 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 19:00:26,085 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000013
2017-04-14 19:00:26,096 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928
2017-04-14 19:00:30,185 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000013] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000013),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:00:31,201 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000013 buffer server: gaming-apex-dhalperi-w-16:43881
2017-04-14 19:00:31,590 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-16]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
Serialization trace:
activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
 context: PTContainer[id=6(container_1492195730173_0001_01_000013),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:00:31,618 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-16]]]]
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
Serialization trace:
activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
	... 2 more
 context: PTContainer[id=6(container_1492195730173_0001_01_000013),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:00:34,104 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000013, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
Container id: container_1492195730173_0001_01_000013
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
	at org.apache.hadoop.util.Shell.run(Shell.java:479)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

2017-04-14 19:00:34,105 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for [e-mail]:52928
2017-04-14 19:00:34,105 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
2017-04-14 19:00:35,127 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
2017-04-14 19:00:35,127 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 19:00:35,127 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[12]]
2017-04-14 19:00:35,127 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[12] on host: [null]
2017-04-14 19:00:36,135 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000014, containerNode=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054, containerNodeURI=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority12
2017-04-14 19:00:36,135 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000013
2017-04-14 19:00:36,138 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000014
2017-04-14 19:00:36,138 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 19:00:36,144 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 19:00:36,144 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000014
2017-04-14 19:00:36,144 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000014 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 19:00:36,144 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000014
2017-04-14 19:00:36,145 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054
2017-04-14 19:00:37,902 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000014] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000014),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:00:38,913 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000014 buffer server: gaming-apex-dhalperi-w-15:59584
2017-04-14 19:00:39,107 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-15]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
Serialization trace:
activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
 context: PTContainer[id=6(container_1492195730173_0001_01_000014),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:00:39,112 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-15]]]]
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
Serialization trace:
activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
	... 2 more
 context: PTContainer[id=6(container_1492195730173_0001_01_000014),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:00:40,160 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000014, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
Container id: container_1492195730173_0001_01_000014
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
	at org.apache.hadoop.util.Shell.run(Shell.java:479)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

2017-04-14 19:00:40,161 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for [e-mail]:53054
2017-04-14 19:00:40,161 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
2017-04-14 19:00:41,199 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
2017-04-14 19:00:41,199 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 19:00:41,199 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[13]]
2017-04-14 19:00:41,199 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[13] on host: [null]
2017-04-14 19:00:42,206 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733
2017-04-14 19:00:42,206 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000015, containerNode=gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733, containerNodeURI=gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority13
2017-04-14 19:00:42,206 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000014
2017-04-14 19:00:42,208 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000015
2017-04-14 19:00:42,208 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 19:00:42,212 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 19:00:42,212 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000015
2017-04-14 19:00:42,212 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000015 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 19:00:42,212 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Set NMClientAsync thread pool size to 21 as the number of nodes to talk to is 11
2017-04-14 19:00:42,214 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000015
2017-04-14 19:00:42,219 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733
2017-04-14 19:00:45,476 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000015] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000015),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:00:46,487 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000015 buffer server: gaming-apex-dhalperi-w-8:57550
2017-04-14 19:00:46,661 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-8]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
Serialization trace:
activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
 context: PTContainer[id=6(container_1492195730173_0001_01_000015),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:00:46,668 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-8]]]]
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
Serialization trace:
activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
	... 2 more
 context: PTContainer[id=6(container_1492195730173_0001_01_000015),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:00:47,225 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000015, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
Container id: container_1492195730173_0001_01_000015
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
	at org.apache.hadoop.util.Shell.run(Shell.java:479)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

2017-04-14 19:00:47,226 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for [e-mail]:53733
2017-04-14 19:00:47,226 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
2017-04-14 19:00:48,263 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
2017-04-14 19:00:48,263 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 19:00:48,264 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[14]]
2017-04-14 19:00:48,264 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[14] on host: [null]
2017-04-14 19:00:49,271 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000016, containerNode=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928, containerNodeURI=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority14
2017-04-14 19:00:49,272 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000015
2017-04-14 19:00:49,274 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000016
2017-04-14 19:00:49,274 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 19:00:49,277 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 19:00:49,277 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000016
2017-04-14 19:00:49,277 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000016 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 19:00:49,278 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000016
2017-04-14 19:00:49,279 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928
2017-04-14 19:00:52,751 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000016] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000016),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:00:53,775 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000016 buffer server: gaming-apex-dhalperi-w-16:52403
2017-04-14 19:00:54,135 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-16]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
Serialization trace:
activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
 context: PTContainer[id=6(container_1492195730173_0001_01_000016),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:00:54,145 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-16]]]]
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
Serialization trace:
activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
	... 2 more
 context: PTContainer[id=6(container_1492195730173_0001_01_000016),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:00:55,291 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000016, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
Container id: container_1492195730173_0001_01_000016
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
	at org.apache.hadoop.util.Shell.run(Shell.java:479)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

2017-04-14 19:00:55,292 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for [e-mail]:52928
2017-04-14 19:00:55,292 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
2017-04-14 19:00:56,328 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
2017-04-14 19:00:56,328 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 19:00:56,329 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[15]]
2017-04-14 19:00:56,329 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[15] on host: [null]
2017-04-14 19:00:57,337 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866
2017-04-14 19:00:57,337 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000017, containerNode=gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866, containerNodeURI=gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority15
2017-04-14 19:00:57,337 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000016
2017-04-14 19:00:57,339 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000017
2017-04-14 19:00:57,340 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 19:00:57,348 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 19:00:57,348 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000017
2017-04-14 19:00:57,348 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000017 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 19:00:57,349 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000017
2017-04-14 19:00:57,354 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866
2017-04-14 19:01:01,038 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000017] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000017),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:01:02,049 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000017 buffer server: gaming-apex-dhalperi-w-14:50842
2017-04-14 19:01:02,198 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-14]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
Serialization trace:
activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
 context: PTContainer[id=6(container_1492195730173_0001_01_000017),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:01:02,204 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-14]]]]
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
Serialization trace:
activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
	... 2 more
 context: PTContainer[id=6(container_1492195730173_0001_01_000017),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:01:03,363 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000017, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
Container id: container_1492195730173_0001_01_000017
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
	at org.apache.hadoop.util.Shell.run(Shell.java:479)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

2017-04-14 19:01:03,363 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for [e-mail]:56866
2017-04-14 19:01:03,364 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]
2017-04-14 19:01:04,387 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]
2017-04-14 19:01:04,387 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null
2017-04-14 19:01:04,388 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[16]]
2017-04-14 19:01:04,388 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[16] on host: [null]
2017-04-14 19:01:05,396 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:43371
2017-04-14 19:01:05,396 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000018, containerNode=gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:43371, containerNodeURI=gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority16
2017-04-14 19:01:05,397 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000017
2017-04-14 19:01:05,399 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000018
2017-04-14 19:01:05,399 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':.
2017-04-14 19:01:05,401 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null}
2017-04-14 19:01:05,401 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts  -Xmx1342177280  for container container_1492195730173_0001_01_000018
2017-04-14 19:01:05,401 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:43371 command: $JAVA_HOME/bin/java  -Xmx1342177280  -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000018 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr  
2017-04-14 19:01:05,402 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000018
2017-04-14 19:01:05,407 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:43371
2017-04-14 19:01:09,197 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000018] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000018),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:01:10,209 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000018 buffer server: gaming-apex-dhalperi-w-6:49764
2017-04-14 19:01:10,377 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-6]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
Serialization trace:
activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
 context: PTContainer[id=6(container_1492195730173_0001_01_000018),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:01:10,385 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-6]]]]
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829)
	at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
	at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
Serialization trace:
activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
	at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
	at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
	at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
	at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
	at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
	at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
	at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
	at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
	... 2 more
 context: PTContainer[id=6(container_1492195730173_0001_01_000018),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]]
2017-04-14 19:01:11,416 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000018, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.
Container id: container_1492195730173_0001_01_000018
Exit code: 1

Issue Links

• relates to: BEAM-2022 ApexTimerInternals seems to treat processing time timers as event time timers
• links to: GitHub Pull Request #2734