Details
-
Bug
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
-
None
Description
Logs below. We tried switching to Java serialization, but that didn't work. We made the field transient (which is broken but let us make progress) and that did.
Stack trace
com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
Serialization trace:
activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
context: PTContainer[id=6(container_1492195730173_0001_01_000012),state=ACTIVE,operators=[
Larger logs with more scope:
2017-04-14 18:56:49,961 INFO com.datatorrent.stram.StreamingAppMaster: Master starting with classpath: ./portability-demo-bundled-apex.jar:/etc/hadoop/conf:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-auth-2.7.3.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-common-2.7.3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7.3.jar:/usr/lib/hadoop/hadoop-nfs-2.7.3.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/hadoop-common-2.7.3.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/snappy-java-1.0.5.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/hadoop-lzo-0.4.19.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/bigquery-connector-0.10.1-hadoop2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/avro-1.7.7.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/gcs-connector-1.6.0-hadoop2.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/zookeeper-3.4.6.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jetty-6.1.26.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/httpclient-4.2.5.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/httpcore-4.2.5.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.5.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/hadoop-auth-2.7.3.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-1.7.4.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7.3.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/hadoop-extras-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-aws-2.7.3.jar:/usr/lib/hadoop-mapreduce/avro-1.7.7.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar:/usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-archives.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/zookeeper-3.4.6.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26.jar:/usr/lib/hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/joda-time-2.9.9.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.42.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix-2.7.3.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/usr/lib/hadoop-mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.7.3.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/httpclient-4.2.5.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar:/usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/jackson-annotations-2.2.3.jar:/usr/lib/hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/jackson-databind-2.2.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.7.3.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/httpcore-4.2.5.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3.jar:/usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3.jar:/usr/lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop-mapreduce/hadoop-aws.jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/jackson-core-2.2.3.jar:/usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/usr/lib/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.5.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.7.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.7.3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/lib/*' 2017-04-14 18:56:49,966 INFO com.datatorrent.stram.StreamingAppMaster: version: 3.5.0 from rev: 6de8828 branch: 6de8828e4f3d5734d0a6f9c1be0aa7057cb60ac8 by Thomas Weise on 06.12.2016 @ 22:11:51 PST 2017-04-14 18:56:49,967 INFO com.datatorrent.stram.StreamingAppMaster: appmaster env: PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games HADOOP_CONF_DIR=/etc/hadoop/conf MAX_APP_ATTEMPTS=2 DATAPROC_MASTER_STANDALONE_COMPONENTS=hadoop-hdfs-secondarynamenode DATAPROC_COMMON_COMPONENTS=openjdk-8-jdk libjansi-java python-numpy libmysql-java hadoop-client hive pig spark-core spark-python spark-r autofs nfs-common libhdfs0 libsnappy1 libatlas3-base libopenblas-base libapr1 vim git bash-completion spark-yarn-shuffle spark-datanucleus spark-extras hadoop-lzo MAIL=/var/mail/yarn LD_LIBRARY_PATH=:/usr/lib/hadoop/lib/native:/usr/lib/hadoop/lib/native HADOOP_HOME_WARN_SUPPRESS=true DATAPROC_WORKER_COMPONENTS=hadoop-hdfs-datanode hadoop-yarn-nodemanager LOGNAME=dhalperi JVM_PID=2123 JSVC_HOME=/usr/lib/bigtop-utils PWD=/hadoop/yarn/nm-local-dir/usercache/dhalperi/appcache/application_1492195730173_0001/container_1492195730173_0001_01_000001 HADOOP_YARN_USER=yarn HADOOP_PREFIX=/usr/lib/hadoop LOCAL_DIRS=/hadoop/yarn/nm-local-dir/usercache/dhalperi/appcache/application_1492195730173_0001 YARN_IDENT_STRING=yarn SHELL=/bin/bash ALPN_JAR=/usr/local/share/google/alpn/alpn-boot-8.1.7.v20160121.jar YARN_CONF_DIR=/etc/hadoop/conf _SYSTEMCTL_SKIP_REDIRECT=true LOG_DIRS=/var/log/hadoop-yarn/userlogs/application_1492195730173_0001/container_1492195730173_0001_01_000001 NM_AUX_SERVICE_mapreduce_shuffle=AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= BDUTIL_DIR=/usr/local/share/google/dataproc/bdutil-dataproc-20170410-124434-RC1 HADOOP_YARN_HOME=/usr/lib/hadoop-yarn YARN_PID_DIR=/var/run/hadoop-yarn DATAPROC_MASTER_EXCLUSIVE_COMPONENTS=hadoop-mapreduce-historyserver hive-metastore hive-server2 nfs-kernel-server spark-history-server NM_AUX_SERVICE_spark_shuffle= HADOOP_HOME=/usr/lib/hadoop SHLVL=5 YARN_ROOT_LOGGER=INFO,RFA JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 LANG=en_US.UTF-8 APP_SUBMIT_TIME_ENV=1492196207488 NM_HOST=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal YARN_LOGFILE=yarn-yarn-nodemanager-gaming-apex-dhalperi-w-15.log HADOOP_USER_NAME=dhalperi DATAPROC_AGENT_JAR=/usr/local/share/google/dataproc/agent-20170410-124434-RC1.jar HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec YARN_LOG_DIR=/var/log/hadoop-yarn HADOOP_HDFS_HOME=/usr/lib/hadoop-hdfs HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce HADOOP_COMMON_HOME=/usr/lib/hadoop _=/usr/lib/jvm/java-8-openjdk-amd64/bin/java APPLICATION_WEB_PROXY_BASE=/proxy/application_1492195730173_0001 NM_HTTP_PORT=8042 NM_PORT=53054 USER=dhalperi CLASSPATH=./*:/etc/hadoop/conf:/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-yarn/*:/usr/lib/hadoop-yarn/lib/*' DATAPROC_MASTER_HA_COMPONENTS=hadoop-hdfs-journalnode hadoop-hdfs-zkfc zookeeper-server HADOOP_TOKEN_FILE_LOCATION=/hadoop/yarn/nm-local-dir/usercache/dhalperi/appcache/application_1492195730173_0001/container_1492195730173_0001_01_000001/container_tokens DATAPROC_MASTER_COMPONENTS=hadoop-hdfs-namenode hadoop-yarn-resourcemanager mysql-server YARN_NICENESS=0 HOME=/home/ CONTAINER_ID=container_1492195730173_0001_01_000001 MALLOC_ARENA_MAX=4 2017-04-14 18:56:50,236 INFO com.datatorrent.stram.StreamingAppMaster: Initializing Application Master. 2017-04-14 18:56:50,311 INFO com.datatorrent.stram.StreamingAppMasterService: Application master, appId=1, clustertimestamp=1492195730173, attemptId=1 2017-04-14 18:56:52,099 INFO com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase: GHFS version: 1.6.0-hadoop2 2017-04-14 18:56:52,491 INFO com.datatorrent.common.util.AsyncFSStorageAgent: using /hadoop/yarn/nm-local-dir/usercache/dhalperi/appcache/application_1492195730173_0001/container_1492195730173_0001_01_000001/tmp/chkp5833413517908828871 as the basepath for checkpointing. 2017-04-14 18:56:56,041 INFO com.datatorrent.stram.FSRecoveryHandler: Creating hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001/recovery/log 2017-04-14 18:56:56,172 INFO com.datatorrent.stram.StreamingAppMasterService: Starting application with 10 operators in 10 containers 2017-04-14 18:56:56,182 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Upper bound of the thread pool size is 500 2017-04-14 18:56:56,183 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 2017-04-14 18:56:56,201 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at gaming-apex-dhalperi-m/10.240.169.55:8030 2017-04-14 18:56:56,224 INFO com.datatorrent.stram.StreamingContainerParent: Config: Configuration: core-default.xml, core-site.xml, yarn-default.xml, yarn-site.xml, mapred-default.xml, mapred-site.xml, hdfs-default.xml, hdfs-site.xml 2017-04-14 18:56:56,224 INFO com.datatorrent.stram.StreamingContainerParent: Listener thread count 30 2017-04-14 18:56:56,230 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue 2017-04-14 18:56:56,235 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 57363 2017-04-14 18:56:56,256 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2017-04-14 18:56:56,257 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 57363: starting 2017-04-14 18:56:56,289 INFO com.datatorrent.stram.StreamingContainerParent: Container callback server listening at gaming-apex-dhalperi-w-15/10.240.169.52:57363 2017-04-14 18:56:56,334 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2017-04-14 18:56:56,417 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2017-04-14 18:56:56,424 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.stram is not defined 2017-04-14 18:56:56,432 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2017-04-14 18:56:56,434 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context stram 2017-04-14 18:56:56,434 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2017-04-14 18:56:56,435 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2017-04-14 18:56:56,437 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /stram/* 2017-04-14 18:56:56,437 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 2017-04-14 18:56:56,658 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 2017-04-14 18:56:56,659 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 32869 2017-04-14 18:57:08,363 WARN com.datatorrent.stram.webapp.TypeGraphFactory: The size of precomputed type graph is 8136 KB 2017-04-14 18:57:08,838 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app stram started at 32869 2017-04-14 18:57:08,838 INFO com.datatorrent.stram.StreamingAppMasterService: Started web service at port: 32869 2017-04-14 18:57:08,839 INFO com.datatorrent.stram.StreamingAppMasterService: Setting tracking URL to: gaming-apex-dhalperi-w-15:32869 2017-04-14 18:57:08,849 INFO com.datatorrent.stram.StreamingAppMasterService: Starting ApplicationMaster 2017-04-14 18:57:08,850 INFO com.datatorrent.stram.StreamingAppMasterService: number of tokens: 1 2017-04-14 18:57:08,908 INFO com.datatorrent.stram.StreamingAppMasterService: Max mem 12288m, Min mem 1024m, Max vcores 32000 and Min vcores 1 capabililty of resources in this cluster 2017-04-14 18:57:08,909 INFO com.datatorrent.stram.StreamingAppMasterService: Blacklist removal time in millis = 3600000, max consecutive node failure count = 2147483647 2017-04-14 18:57:08,913 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at gaming-apex-dhalperi-m/10.240.169.55:8032 2017-04-14 18:57:10,127 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=9,name=SumTeamScores/GroupByKey] 2017-04-14 18:57:10,127 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=1,name=TextIO.Read/Read] 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=4,name=FixedWindows/Window.Assign] 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)] 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=2,name=ParseGameEvent/ParMultiDo(ParseEvent)] 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] 2017-04-14 18:57:10,131 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=5,name=SumTeamScores/ParDo(KeyScoreByTeam)/ParMultiDo(KeyScoreByTeam)] 2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)] 2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=3,name=SetTimestamps/ParMultiDo(SetTimestamps)] 2017-04-14 18:57:10,132 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 18:57:10,132 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[0], Capability[<memory:1536, vCores:1>]Priority[1], Capability[<memory:1536, vCores:1>]Priority[2], Capability[<memory:1024, vCores:1>]Priority[3], Capability[<memory:1536, vCores:1>]Priority[4], Capability[<memory:1536, vCores:1>]Priority[5], Capability[<memory:1536, vCores:1>]Priority[6], Capability[<memory:1536, vCores:1>]Priority[7], Capability[<memory:1536, vCores:1>]Priority[8], Capability[<memory:1536, vCores:1>]Priority[9]] 2017-04-14 18:57:10,132 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[0] on host: [null] 2017-04-14 18:57:10,137 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[1] on host: [null] 2017-04-14 18:57:10,137 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[2] on host: [null] 2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1024, vCores:1>]Priority[3] on host: [null] 2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[4] on host: [null] 2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[5] on host: [null] 2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[6] on host: [null] 2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[7] on host: [null] 2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[8] on host: [null] 2017-04-14 18:57:10,138 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[9] on host: [null] 2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:37663 2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:50931 2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:33276 2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:37140 2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:55762 2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743 2017-04-14 18:57:11,216 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:46370 2017-04-14 18:57:11,217 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054 2017-04-14 18:57:11,217 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:39804 2017-04-14 18:57:11,218 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000002, containerNode=gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:37663, containerNodeURI=gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority0 2017-04-14 18:57:11,245 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000002 2017-04-14 18:57:11,251 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 18:57:11,307 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 18:57:11,307 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000002 2017-04-14 18:57:11,307 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:37663 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000002 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 18:57:11,317 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000003, containerNode=gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:50931, containerNodeURI=gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority1 2017-04-14 18:57:11,321 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000002 2017-04-14 18:57:11,325 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000003 2017-04-14 18:57:11,326 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 18:57:11,332 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 18:57:11,332 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000003 2017-04-14 18:57:11,334 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:50931 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000003 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 18:57:11,334 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-9.c.clouddfe.google.com.internal:37663 2017-04-14 18:57:11,339 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000004, containerNode=gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:33276, containerNodeURI=gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority2 2017-04-14 18:57:11,342 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000003 2017-04-14 18:57:11,345 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000004 2017-04-14 18:57:11,346 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 18:57:11,350 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 18:57:11,351 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000004 2017-04-14 18:57:11,351 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:33276 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000004 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 18:57:11,352 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000005, containerNode=gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:37140, containerNodeURI=gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:8042, containerResourceMemory1024, priority3 2017-04-14 18:57:11,359 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000005 2017-04-14 18:57:11,360 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 18:57:11,361 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000004 2017-04-14 18:57:11,371 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 18:57:11,372 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx805306368 for container container_1492195730173_0001_01_000005 2017-04-14 18:57:11,372 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:37140 command: $JAVA_HOME/bin/java -Xmx805306368 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000005 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 18:57:11,373 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000006, containerNode=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928, containerNodeURI=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority4 2017-04-14 18:57:11,373 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000005 2017-04-14 18:57:11,384 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000006 2017-04-14 18:57:11,384 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 18:57:11,387 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 18:57:11,388 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000006 2017-04-14 18:57:11,388 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000006 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 18:57:11,389 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000007, containerNode=gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:55762, containerNodeURI=gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority5 2017-04-14 18:57:11,394 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000007 2017-04-14 18:57:11,394 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 18:57:11,401 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000006 2017-04-14 18:57:11,401 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 18:57:11,406 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000007 2017-04-14 18:57:11,407 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:55762 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000007 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 18:57:11,407 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000008, containerNode=gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743, containerNodeURI=gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority6 2017-04-14 18:57:11,412 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-1.c.clouddfe.google.com.internal:37140 2017-04-14 18:57:11,413 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000007 2017-04-14 18:57:11,415 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000008 2017-04-14 18:57:11,415 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 18:57:11,418 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-5.c.clouddfe.google.com.internal:33276 2017-04-14 18:57:11,421 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 18:57:11,422 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000008 2017-04-14 18:57:11,422 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000008 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 18:57:11,422 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000009, containerNode=gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:46370, containerNodeURI=gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority7 2017-04-14 18:57:11,426 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000008 2017-04-14 18:57:11,430 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000009 2017-04-14 18:57:11,430 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 18:57:11,433 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-11.c.clouddfe.google.com.internal:50931 2017-04-14 18:57:11,436 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 18:57:11,436 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000009 2017-04-14 18:57:11,436 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:46370 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000009 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 18:57:11,437 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000010, containerNode=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054, containerNodeURI=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority8 2017-04-14 18:57:11,437 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000009 2017-04-14 18:57:11,449 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743 2017-04-14 18:57:11,449 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000010 2017-04-14 18:57:11,450 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 18:57:11,454 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-12.c.clouddfe.google.com.internal:55762 2017-04-14 18:57:11,454 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 18:57:11,455 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000010 2017-04-14 18:57:11,455 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000010 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 18:57:11,455 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000011, containerNode=gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:39804, containerNodeURI=gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority9 2017-04-14 18:57:11,459 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 2017-04-14 18:57:11,463 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000011 2017-04-14 18:57:11,463 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 18:57:11,460 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000010 2017-04-14 18:57:11,473 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-2.c.clouddfe.google.com.internal:46370 2017-04-14 18:57:11,474 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054 2017-04-14 18:57:11,482 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 18:57:11,482 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000011 2017-04-14 18:57:11,482 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:39804 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000011 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 18:57:11,490 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000011 2017-04-14 18:57:11,500 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-18.c.clouddfe.google.com.internal:39804 2017-04-14 18:57:13,756 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000010] Entering heartbeat loop.. context: PTContainer[id=7(container_1492195730173_0001_01_000010),state=ALLOCATED,operators=[PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)]]] 2017-04-14 18:57:14,770 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000010 buffer server: gaming-apex-dhalperi-w-15:55080 2017-04-14 18:57:14,941 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000008] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000008),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 18:57:15,054 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000002] Entering heartbeat loop.. context: PTContainer[id=9(container_1492195730173_0001_01_000002),state=ALLOCATED,operators=[PTOperator[id=9,name=SumTeamScores/GroupByKey]]] 2017-04-14 18:57:15,073 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000009] Entering heartbeat loop.. context: PTContainer[id=5(container_1492195730173_0001_01_000009),state=ALLOCATED,operators=[PTOperator[id=5,name=SumTeamScores/ParDo(KeyScoreByTeam)/ParMultiDo(KeyScoreByTeam)]]] 2017-04-14 18:57:15,106 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000003] Entering heartbeat loop.. context: PTContainer[id=1(container_1492195730173_0001_01_000003),state=ALLOCATED,operators=[PTOperator[id=1,name=TextIO.Read/Read]]] 2017-04-14 18:57:15,125 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000011] Entering heartbeat loop.. context: PTContainer[id=3(container_1492195730173_0001_01_000011),state=ALLOCATED,operators=[PTOperator[id=3,name=SetTimestamps/ParMultiDo(SetTimestamps)]]] 2017-04-14 18:57:15,232 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000006] Entering heartbeat loop.. context: PTContainer[id=2(container_1492195730173_0001_01_000006),state=ALLOCATED,operators=[PTOperator[id=2,name=ParseGameEvent/ParMultiDo(ParseEvent)]]] 2017-04-14 18:57:15,333 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000007] Entering heartbeat loop.. context: PTContainer[id=8(container_1492195730173_0001_01_000007),state=ALLOCATED,operators=[PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)]]] 2017-04-14 18:57:15,346 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000004] Entering heartbeat loop.. context: PTContainer[id=4(container_1492195730173_0001_01_000004),state=ALLOCATED,operators=[PTOperator[id=4,name=FixedWindows/Window.Assign]]] 2017-04-14 18:57:15,472 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000005] Entering heartbeat loop.. context: PTContainer[id=10(container_1492195730173_0001_01_000005),state=ALLOCATED,operators=[PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]]] 2017-04-14 18:57:15,954 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000008 buffer server: gaming-apex-dhalperi-w-3:60148 2017-04-14 18:57:16,066 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000002 buffer server: gaming-apex-dhalperi-w-9:52966 2017-04-14 18:57:16,085 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000009 buffer server: gaming-apex-dhalperi-w-2:41280 2017-04-14 18:57:16,116 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000003 buffer server: gaming-apex-dhalperi-w-11:41089 2017-04-14 18:57:16,136 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000011 buffer server: gaming-apex-dhalperi-w-18:50437 2017-04-14 18:57:16,244 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000006 buffer server: gaming-apex-dhalperi-w-16:55760 2017-04-14 18:57:16,345 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000007 buffer server: gaming-apex-dhalperi-w-12:42777 2017-04-14 18:57:16,359 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000004 buffer server: gaming-apex-dhalperi-w-5:58095 2017-04-14 18:57:16,485 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000005 buffer server: gaming-apex-dhalperi-w-1:55004 2017-04-14 19:00:14,948 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196414947, last window id change time 1492196354898, window processing timeout millis 60000 2017-04-14 19:00:14,948 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=9,name=SumTeamScores/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196414947, last window id change time 1492196354896, window processing timeout millis 60000 2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196355249, window processing timeout millis 60000 2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196355318, window processing timeout millis 60000 2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196354898, window processing timeout millis 60000 2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=9,name=SumTeamScores/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196354896, window processing timeout millis 60000 2017-04-14 19:00:15,950 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196415950, last window id change time 1492196354966, window processing timeout millis 60000 2017-04-14 19:00:15,951 INFO com.datatorrent.stram.StreamingContainerManager: Blocked operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] container PTContainer[id=6(container_1492195730173_0001_01_000008),state=ACTIVE] time 60701ms 2017-04-14 19:00:16,951 INFO com.datatorrent.stram.StreamingAppMasterService: Requested stop container container_1492195730173_0001_01_000008 2017-04-14 19:00:16,951 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: STOP_CONTAINER for Container container_1492195730173_0001_01_000008 2017-04-14 19:00:16,953 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196355249, window processing timeout millis 60000 2017-04-14 19:00:16,953 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196355318, window processing timeout millis 60000 2017-04-14 19:00:16,954 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196354898, window processing timeout millis 60000 2017-04-14 19:00:16,954 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=9,name=SumTeamScores/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196354896, window processing timeout millis 60000 2017-04-14 19:00:16,954 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196416953, last window id change time 1492196354966, window processing timeout millis 60000 2017-04-14 19:00:16,954 INFO com.datatorrent.stram.StreamingContainerManager: Blocked operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] container PTContainer[id=6(container_1492195730173_0001_01_000008),state=ACTIVE] time 61704ms 2017-04-14 19:00:16,957 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743 2017-04-14 19:00:17,962 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000008, state=COMPLETE, exitStatus=-105, diagnostics=Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2017-04-14 19:00:17,962 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000008@gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743 2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196355249, window processing timeout millis 60000 2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196355318, window processing timeout millis 60000 2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196354898, window processing timeout millis 60000 2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=9,name=SumTeamScores/GroupByKey] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196354896, window processing timeout millis 60000 2017-04-14 19:00:17,965 WARN com.datatorrent.stram.StreamingContainerManager: Marking operator PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)] blocked committed window 58f11b740000003b, recovery window 58f11b740000003b, current time 1492196417965, last window id change time 1492196354966, window processing timeout millis 60000 2017-04-14 19:00:17,965 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]] 2017-04-14 19:00:19,013 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] 2017-04-14 19:00:19,013 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 19:00:19,014 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[10]] 2017-04-14 19:00:19,014 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[10] on host: [null] 2017-04-14 19:00:20,021 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000012, containerNode=gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743, containerNodeURI=gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority10 2017-04-14 19:00:20,022 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000008 2017-04-14 19:00:20,024 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000012 2017-04-14 19:00:20,024 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 19:00:20,034 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 19:00:20,035 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000012 2017-04-14 19:00:20,035 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 19:00:20,035 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000012 2017-04-14 19:00:20,036 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743 2017-04-14 19:00:21,762 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000012] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000012),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:00:22,774 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000012 buffer server: gaming-apex-dhalperi-w-3:54029 2017-04-14 19:00:23,006 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-3]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData Serialization trace: activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator) at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192) at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137) at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914) at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862) at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820) at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705) at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310) context: PTContainer[id=6(container_1492195730173_0001_01_000012),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:00:23,011 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-3]]]] at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829) at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705) at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310) Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData Serialization trace: activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator) at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192) at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137) at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914) at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862) at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820) ... 2 more context: PTContainer[id=6(container_1492195730173_0001_01_000012),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:00:24,048 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000012, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch. Container id: container_1492195730173_0001_01_000012 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:582) at org.apache.hadoop.util.Shell.run(Shell.java:479) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 2017-04-14 19:00:24,049 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000012@gaming-apex-dhalperi-w-3.c.clouddfe.google.com.internal:44743 2017-04-14 19:00:24,049 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]] 2017-04-14 19:00:25,072 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] 2017-04-14 19:00:25,072 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 19:00:25,072 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[11]] 2017-04-14 19:00:25,072 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[11] on host: [null] 2017-04-14 19:00:26,078 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000013, containerNode=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928, containerNodeURI=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority11 2017-04-14 19:00:26,079 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000012 2017-04-14 19:00:26,081 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000013 2017-04-14 19:00:26,081 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 19:00:26,084 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 19:00:26,084 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000013 2017-04-14 19:00:26,085 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000013 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 19:00:26,085 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000013 2017-04-14 19:00:26,096 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 2017-04-14 19:00:30,185 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000013] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000013),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:00:31,201 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000013 buffer server: gaming-apex-dhalperi-w-16:43881 2017-04-14 19:00:31,590 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-16]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData Serialization trace: activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator) at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192) at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137) at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914) at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862) at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820) at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705) at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310) context: PTContainer[id=6(container_1492195730173_0001_01_000013),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:00:31,618 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-16]]]] at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829) at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705) at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310) Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData Serialization trace: activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator) at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192) at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137) at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914) at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862) at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820) ... 2 more context: PTContainer[id=6(container_1492195730173_0001_01_000013),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:00:34,104 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000013, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch. Container id: container_1492195730173_0001_01_000013 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:582) at org.apache.hadoop.util.Shell.run(Shell.java:479) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 2017-04-14 19:00:34,105 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000013@gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 2017-04-14 19:00:34,105 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]] 2017-04-14 19:00:35,127 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] 2017-04-14 19:00:35,127 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 19:00:35,127 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[12]] 2017-04-14 19:00:35,127 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[12] on host: [null] 2017-04-14 19:00:36,135 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000014, containerNode=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054, containerNodeURI=gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority12 2017-04-14 19:00:36,135 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000013 2017-04-14 19:00:36,138 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000014 2017-04-14 19:00:36,138 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 19:00:36,144 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 19:00:36,144 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000014 2017-04-14 19:00:36,144 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000014 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 19:00:36,144 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000014 2017-04-14 19:00:36,145 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054 2017-04-14 19:00:37,902 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000014] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000014),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:00:38,913 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000014 buffer server: gaming-apex-dhalperi-w-15:59584 2017-04-14 19:00:39,107 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-15]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData Serialization trace: activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator) at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192) at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137) at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914) at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862) at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820) at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705) at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310) context: PTContainer[id=6(container_1492195730173_0001_01_000014),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:00:39,112 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-15]]]] at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829) at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705) at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310) Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData Serialization trace: activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator) at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192) at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137) at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914) at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862) at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820) ... 2 more context: PTContainer[id=6(container_1492195730173_0001_01_000014),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:00:40,160 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000014, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch. Container id: container_1492195730173_0001_01_000014 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:582) at org.apache.hadoop.util.Shell.run(Shell.java:479) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 2017-04-14 19:00:40,161 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000014@gaming-apex-dhalperi-w-15.c.clouddfe.google.com.internal:53054 2017-04-14 19:00:40,161 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]] 2017-04-14 19:00:41,199 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] 2017-04-14 19:00:41,199 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 19:00:41,199 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[13]] 2017-04-14 19:00:41,199 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[13] on host: [null] 2017-04-14 19:00:42,206 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733 2017-04-14 19:00:42,206 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000015, containerNode=gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733, containerNodeURI=gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority13 2017-04-14 19:00:42,206 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000014 2017-04-14 19:00:42,208 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000015 2017-04-14 19:00:42,208 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 19:00:42,212 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 19:00:42,212 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000015 2017-04-14 19:00:42,212 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000015 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 19:00:42,212 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Set NMClientAsync thread pool size to 21 as the number of nodes to talk to is 11 2017-04-14 19:00:42,214 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000015 2017-04-14 19:00:42,219 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733 2017-04-14 19:00:45,476 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000015] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000015),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:00:46,487 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000015 buffer server: gaming-apex-dhalperi-w-8:57550 2017-04-14 19:00:46,661 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-8]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData Serialization trace: activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator) at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192) at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137) at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914) at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862) at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820) at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705) at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310) context: PTContainer[id=6(container_1492195730173_0001_01_000015),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:00:46,668 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-8]]]] at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829) at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705) at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310) Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData Serialization trace: activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator) at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192) at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137) at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914) at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862) at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820) ... 2 more context: PTContainer[id=6(container_1492195730173_0001_01_000015),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:00:47,225 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000015, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch. Container id: container_1492195730173_0001_01_000015 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:582) at org.apache.hadoop.util.Shell.run(Shell.java:479) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 2017-04-14 19:00:47,226 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000015@gaming-apex-dhalperi-w-8.c.clouddfe.google.com.internal:53733 2017-04-14 19:00:47,226 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]] 2017-04-14 19:00:48,263 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] 2017-04-14 19:00:48,263 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 19:00:48,264 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[14]] 2017-04-14 19:00:48,264 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[14] on host: [null] 2017-04-14 19:00:49,271 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000016, containerNode=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928, containerNodeURI=gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority14 2017-04-14 19:00:49,272 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000015 2017-04-14 19:00:49,274 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000016 2017-04-14 19:00:49,274 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 19:00:49,277 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 19:00:49,277 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000016 2017-04-14 19:00:49,277 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000016 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 19:00:49,278 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000016 2017-04-14 19:00:49,279 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 2017-04-14 19:00:52,751 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000016] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000016),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:00:53,775 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000016 buffer server: gaming-apex-dhalperi-w-16:52403 2017-04-14 19:00:54,135 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-16]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData Serialization trace: activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator) at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192) at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137) at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914) at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862) at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820) at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705) at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310) context: PTContainer[id=6(container_1492195730173_0001_01_000016),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:00:54,145 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-16]]]] at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829) at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705) at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310) Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData Serialization trace: activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator) at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192) at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137) at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914) at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862) at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820) ... 2 more context: PTContainer[id=6(container_1492195730173_0001_01_000016),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:00:55,291 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000016, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch. Container id: container_1492195730173_0001_01_000016 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:582) at org.apache.hadoop.util.Shell.run(Shell.java:479) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 2017-04-14 19:00:55,292 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000016@gaming-apex-dhalperi-w-16.c.clouddfe.google.com.internal:52928 2017-04-14 19:00:55,292 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]] 2017-04-14 19:00:56,328 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] 2017-04-14 19:00:56,328 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 19:00:56,329 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[15]] 2017-04-14 19:00:56,329 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[15] on host: [null] 2017-04-14 19:00:57,337 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866 2017-04-14 19:00:57,337 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000017, containerNode=gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866, containerNodeURI=gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority15 2017-04-14 19:00:57,337 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000016 2017-04-14 19:00:57,339 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000017 2017-04-14 19:00:57,340 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 19:00:57,348 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 19:00:57,348 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000017 2017-04-14 19:00:57,348 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000017 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 19:00:57,349 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000017 2017-04-14 19:00:57,354 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866 2017-04-14 19:01:01,038 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000017] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000017),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:01:02,049 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000017 buffer server: gaming-apex-dhalperi-w-14:50842 2017-04-14 19:01:02,198 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-14]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData Serialization trace: activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator) at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192) at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137) at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914) at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862) at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820) at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705) at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310) context: PTContainer[id=6(container_1492195730173_0001_01_000017),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:01:02,204 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-14]]]] at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829) at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705) at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310) Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData Serialization trace: activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator) at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192) at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137) at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914) at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862) at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820) ... 2 more context: PTContainer[id=6(container_1492195730173_0001_01_000017),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:01:03,363 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000017, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch. Container id: container_1492195730173_0001_01_000017 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:582) at org.apache.hadoop.util.Shell.run(Shell.java:479) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 2017-04-14 19:01:03,363 INFO com.datatorrent.stram.StreamingContainerManager: Initiating recovery for container_1492195730173_0001_01_000017@gaming-apex-dhalperi-w-14.c.clouddfe.google.com.internal:56866 2017-04-14 19:01:03,364 INFO com.datatorrent.stram.StreamingContainerManager: Affected operators [PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey], PTOperator[id=7,name=SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)], PTOperator[id=8,name=SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow)], PTOperator[id=9,name=SumTeamScores/GroupByKey], PTOperator[id=10,name=SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles)]] 2017-04-14 19:01:04,387 INFO com.datatorrent.stram.ResourceRequestHandler: Strict anti-affinity = [] for container with operators PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey] 2017-04-14 19:01:04,387 INFO com.datatorrent.stram.ResourceRequestHandler: Found host null 2017-04-14 19:01:04,388 INFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Capability[<memory:1536, vCores:1>]Priority[16]] 2017-04-14 19:01:04,388 INFO com.datatorrent.stram.StreamingAppMasterService: Requested container: Capability[<memory:1536, vCores:1>]Priority[16] on host: [null] 2017-04-14 19:01:05,396 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Received new token for : gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:43371 2017-04-14 19:01:05,396 INFO com.datatorrent.stram.StreamingAppMasterService: Got new container., containerId=container_1492195730173_0001_01_000018, containerNode=gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:43371, containerNodeURI=gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:8042, containerResourceMemory2048, priority16 2017-04-14 19:01:05,397 INFO com.datatorrent.stram.StreamingContainerManager: Removing container agent container_1492195730173_0001_01_000017 2017-04-14 19:01:05,399 INFO com.datatorrent.stram.LaunchContainerRunnable: Setting up container launch context for containerid=container_1492195730173_0001_01_000018 2017-04-14 19:01:05,399 INFO com.datatorrent.stram.LaunchContainerRunnable: CLASSPATH: ./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*':. 2017-04-14 19:01:05,401 INFO com.datatorrent.common.util.BasicContainerOptConfigurator: property map for operator {-Xmx=768m, Generic=null} 2017-04-14 19:01:05,401 INFO com.datatorrent.stram.LaunchContainerRunnable: Jvm opts -Xmx1342177280 for container container_1492195730173_0001_01_000018 2017-04-14 19:01:05,401 INFO com.datatorrent.stram.LaunchContainerRunnable: Launching on node: gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:43371 command: $JAVA_HOME/bin/java -Xmx1342177280 -Ddt.attr.APPLICATION_PATH=hdfs://gaming-apex-dhalperi-m/user/dhalperi/datatorrent/apps/application_1492195730173_0001 -Djava.io.tmpdir=$PWD/tmp -Ddt.cid=container_1492195730173_0001_01_000018 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=<LOG_DIR> com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr 2017-04-14 19:01:05,402 INFO org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1492195730173_0001_01_000018 2017-04-14 19:01:05,407 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : gaming-apex-dhalperi-w-6.c.clouddfe.google.com.internal:43371 2017-04-14 19:01:09,197 INFO com.datatorrent.stram.StreamingContainerParent: child msg: [container_1492195730173_0001_01_000018] Entering heartbeat loop.. context: PTContainer[id=6(container_1492195730173_0001_01_000018),state=ALLOCATED,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:01:10,209 INFO com.datatorrent.stram.StreamingContainerManager: Container container_1492195730173_0001_01_000018 buffer server: gaming-apex-dhalperi-w-6:49764 2017-04-14 19:01:10,377 INFO com.datatorrent.stram.StreamingContainerParent: child msg: deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-6]]]] com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData Serialization trace: activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator) at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192) at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137) at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914) at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862) at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820) at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705) at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310) context: PTContainer[id=6(container_1492195730173_0001_01_000018),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:01:10,385 INFO com.datatorrent.stram.StreamingContainerParent: child msg: java.lang.IllegalStateException: Deploy request failed: [OperatorDeployInfo[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey,type=GENERIC,checkpoint={58f11b740000003b, 0, 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=stream7,sourceNodeId=5,sourcePortName=output,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream4,bufferServer=gaming-apex-dhalperi-w-6]]]] at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:829) at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705) at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310) Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData Serialization trace: activeTimers (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator) at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192) at com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137) at com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914) at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862) at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820) ... 2 more context: PTContainer[id=6(container_1492195730173_0001_01_000018),state=ACTIVE,operators=[PTOperator[id=6,name=SumTeamScores/Combine.perKey(SumInteger)/GroupByKey]]] 2017-04-14 19:01:11,416 INFO com.datatorrent.stram.StreamingAppMasterService: Completed containerId=container_1492195730173_0001_01_000018, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch. Container id: container_1492195730173_0001_01_000018 Exit code: 1
Attachments
Issue Links
- relates to
-
BEAM-2022 ApexTimerInternals seems to treat processing time timers as event time timers
- Resolved
- links to