[hadoop@ip-172-31-90-92 ~]$ yarn logs -applicationId application_1555406634668_0016 19/04/16 12:01:05 INFO client.RMProxy: Connecting to ResourceManager at ip-172-31-90-92.ec2.internal/172.31.90.92:8032 Container: container_1555406634668_0016_01_000002 on ip-172-31-91-60.ec2.internal_8041 ======================================================================================== LogType:stderr Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:188 Log Contents: 2019-04-16 11:51:50 Starting to run new task attempt: attempt_1555406634668_0016_1_00_000000_0 2019-04-16 11:51:52 Completed running task attempt: attempt_1555406634668_0016_1_00_000000_0 End of LogType:stderr LogType:stdout Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:188 Log Contents: 2019-04-16 11:51:50 Starting to run new task attempt: attempt_1555406634668_0016_1_00_000000_0 2019-04-16 11:51:52 Completed running task attempt: attempt_1555406634668_0016_1_00_000000_0 End of LogType:stdout LogType:syslog Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:39826 Log Contents: 2019-04-16 11:51:49,358 [INFO] [main] |task.TezChild|: TezChild starting with PID=17176, containerIdentifier=container_1555406634668_0016_01_000002 2019-04-16 11:51:49,804 [INFO] [main] |shim.HadoopShimsLoader|: Trying to locate HadoopShimProvider for hadoopVersion=2.8.5-amzn-1, majorVersion=2, minorVersion=8 2019-04-16 11:51:49,805 [INFO] [main] |shim.HadoopShimsLoader|: Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.8.5-amzn-1, majorVersion=2, minorVersion=8 2019-04-16 11:51:49,834 [INFO] [main] |task.TezChild|: /************************************************************ [system properties] os.name: Linux os.version: 4.14.77-70.82.amzn1.x86_64 java.home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.42.amzn1.x86_64/jre java.runtime.version: 1.8.0_191-b12 java.vendor: Oracle Corporation java.version: 1.8.0_191 java.vm.name: OpenJDK 64-Bit Server VM java.class.path: /mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/emr-ddb-hive.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/emr-kinesis-hive.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/hive-openx-serde.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/emr-hive-goodies.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/hive-hcatalog-server-extensions-2.3.4-amzn-0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/emr-s3-select-hive-connector.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/hive-exec-2.3.4-amzn-0-b4b465141bab0a22bba33ac51550e7bd41a2a42cf3dd3f440846c25eb6e4c431.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/hive-hcatalog-pig-adapter-2.3.4-amzn-0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/hive-hcatalog-core-2.3.4-amzn-0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/aws-glue-datacatalog-hive2-client.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/hive-hcatalog-streaming-2.3.4-amzn-0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/tez-yarn-timeline-history-with-acls-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/tez-tests-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/tez-mapreduce-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/tez-ext-service-tests-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/tez-runtime-library-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/tez-dag-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/hadoop-shim-2.7-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/tez-history-parser-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/tez-runtime-internals-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/tez-job-analyzer-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/tez-examples-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/hadoop-shim-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/tez-common-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/tez-javadoc-tools-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/tez-yarn-timeline-history-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/tez-api-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/commons-cli-1.2.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/hadoop-mapreduce-client-core-2.8.5-amzn-1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/commons-math3-3.1.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/commons-collections-3.2.2.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/guava-11.0.2.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/commons-lang-2.6.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/slf4j-api-1.7.10.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/netty-3.6.2.Final.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/commons-codec-1.4.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/commons-io-2.4.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/async-http-client-1.8.16.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/RoaringBitmap-0.4.9.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/jettison-1.3.4.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/jetty-util-6.1.26.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/jsr305-3.0.0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/jetty-6.1.26.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/servlet-api-2.5.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/jersey-client-1.9.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/metrics-core-3.1.0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/commons-collections4-4.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/protobuf-java-2.5.0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/hadoop-mapreduce-client-common-2.8.5-amzn-1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/jersey-json-1.9.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/hadoop-mapreduce-client-common.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/hhadoop-mapreduce-client-core.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/hadoop-auth.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/hadoop-annotations.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tezlib/lib/hadoop-yarn-server-web-proxy.jar:/etc/hadoop/conf:/usr/lib/hadoop/hadoop-rumen-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-distcp-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-azure-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/hadoop-streaming-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-openstack-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/hadoop/hadoop-openstack.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-sls-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-extras-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-datajoin-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-auth-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-archives.jar:/usr/lib/hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/hadoop-annotations-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/hadoop/hadoop-common-2.8.5-amzn-1-tests.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-gridmix-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-ant-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-nfs-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-aws-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-streaming.jar:/usr/lib/hadoop/hadoop-sls.jar:/usr/lib/hadoop/hadoop-common-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-azure-datalake.jar:/usr/lib/hadoop/hadoop-archives-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.5-amzn-1.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/nimbus-jose-jwt-4.41.1.jar:/usr/lib/hadoop/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/audience-annotations-0.5.0.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/jcip-annotations-1.0-1.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/json-smart-1.3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/httpcore-4.4.9.jar:/usr/lib/hadoop/lib/zookeeper-3.4.13.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/httpclient-4.5.5.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/jsch-0.1.54.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.5-amzn-1-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.5-amzn-1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.5-amzn-1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.5-amzn-1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.8.5-amzn-1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.5-amzn-1-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.5-amzn-1-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/okio-1.4.0.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/okhttp-2.4.0.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/nimbus-jose-jwt-4.41.1.jar:/usr/lib/hadoop-mapreduce/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/azure-data-lake-store-sdk-2.2.3.jar:/usr/lib/hadoop-mapreduce/commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/audience-annotations-0.5.0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/jcip-annotations-1.0-1.jar:/usr/lib/hadoop-mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-core-1.11.461.jar:/usr/lib/hadoop-mapreduce/json-smart-1.3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/httpcore-4.4.9.jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.7.jar:/usr/lib/hadoop-mapreduce/zookeeper-3.4.13.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-extras-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-auth-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-archives.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/httpclient-4.5.5.jar:/usr/lib/hadoop-mapreduce/joda-time-2.9.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/okio-1.4.0.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.54.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/azure-storage-2.2.0.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.5-amzn-1-tests.jar:/usr/lib/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/jackson-databind-2.6.7.2.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/okhttp-2.4.0.jar:/usr/lib/hadoop-mapreduce/jmespath-java-1.11.461.jar:/usr/lib/hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/hadoop-ant-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/jackson-dataformat-cbor-2.6.7.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/jackson-annotations-2.6.7.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs.jar:/usr/lib/hadoop-mapreduce/hadoop-aws-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-s3-1.11.461.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-kms-1.11.461.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/lib/curator-test-2.7.1.jar:/usr/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.13-tests.jar:/usr/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/lib/hadoop-yarn/lib/audience-annotations-0.5.0.jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.13.jar:/usr/lib/hadoop-yarn/lib/commons-math-2.2.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/lib/hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/lib/hadoop-yarn/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.11.393.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/jcl-over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.29.0.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar:/usr/lib/spark/yarn/lib/datanucleus-core.jar:/usr/lib/spark/yarn/lib/datanucleus-rdbms.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.1.0.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticsearch-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-storagegateway-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-clouddirectory-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacemeteringservice-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-signer-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-snowball-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticbeanstalk-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitosync-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloud9-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3control-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastoredata-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscalingplans-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediapackage-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-secretsmanager-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-transcribe-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-applicationautoscaling-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicediscovery-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediaconvert-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-licensemanager-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplaceentitlement-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-translate-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticache-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-transfer-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-migrationhub-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidentity-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-fms-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-machinelearning-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directory-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elastictranscoder-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpleworkflow-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacecommerceanalytics-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastore-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-xray-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dynamodb-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-connect-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-macie-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lexmodelbuilding-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpointemail-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesisvideo-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ram-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicecatalog-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-serverlessapplicationrepository-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidp-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesisanalyticsv2-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-support-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mq-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-comprehendmedical-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecr-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-waf-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediatailor-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-eks-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-robomaker-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workmail-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costandusagereport-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-stepfunctions-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-guardduty-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-discovery-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpoint-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mobile-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workdocs-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glue-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-code-generator-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-medialive-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot1clickdevices-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-chime-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-organizations-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemakerruntime-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dlm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudformation-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rekognition-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot1clickprojects-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-health-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appmesh-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworkscm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sns-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-greengrass-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servermigration-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-fsx-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-logs-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api-gateway-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-alexaforbusiness-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pi-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-neptune-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-batch-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudtrail-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-amplify-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glacier-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-datasync-1.11.461.jar:/usr/share/aws/aws-java-sdk/jmespath-java-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-quicksight-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-budgets-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemaker-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rdsdata-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dax-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-globalaccelerator-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancing-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroups-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kafka-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecs-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iotanalytics-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpointsmsvoice-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iotjobsdataplane-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworks-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsmv2-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesis-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-datapipeline-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pricing-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lightsail-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediaconnect-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-comprehend-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-devicefarm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpledb-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudfront-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acmpca-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53resolver-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancingv2-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codepipeline-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-events-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codestar-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatch-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costexplorer-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appsync-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iam-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-securityhub-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kms-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11.461.jar: java.io.tmpdir: /mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002/tmp user.dir: /mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000002 user.name: yarn ************************************************************/ 2019-04-16 11:51:49,875 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_1555406634668_0016_01_000002 2019-04-16 11:51:50,024 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_1555406634668_0016_01_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1555406634668_0016_1_00_000000_0 2019-04-16 11:51:50,026 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: attempt_1555406634668_0016_1_00_000000_0 2019-04-16 11:51:52,126 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_1555406634668_0016_01_000002 2019-04-16 11:52:00,284 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_1555406634668_0016_01_000002: 8157 ms after starting to poll. TaskInfo: shouldDie: true 2019-04-16 11:52:00,284 [INFO] [main] |task.TezChild|: ContainerTask returned shouldDie=true for container container_1555406634668_0016_01_000002, Exiting 2019-04-16 11:52:00,285 [INFO] [main] |task.TezChild|: Shutdown invoked for container container_1555406634668_0016_01_000002 2019-04-16 11:52:00,285 [INFO] [main] |task.TezChild|: Shutting down container container_1555406634668_0016_01_000002 End of LogType:syslog LogType:syslog_attempt_1555406634668_0016_1_00_000000_0 Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:12986 Log Contents: 2019-04-16 11:51:50,032 [INFO] [main] |task.TezChild|: Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0 2019-04-16 11:51:50,051 [INFO] [main] |metrics.TaskCounterUpdater|: Using ResourceCalculatorProcessTree : org.apache.tez.util.TezMxBeanResourceCalculator 2019-04-16 11:51:50,051 [INFO] [main] |runtime.LogicalIOProcessorRuntimeTask|: Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : insert overwrite table idTable values (1)(Stage-1), VertexName: Map 1, VertexParallelism: 1, TaskAttemptID:attempt_1555406634668_0016_1_00_000000_0, processorName=org.apache.hadoop.hive.ql.exec.tez.MapTezProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=values__tmp__table__1, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInputLegacy }}, ], outputSpecList=[{{ destinationVertexName=out_Map 1, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ], taskConfEntryCount=0 2019-04-16 11:51:50,053 [INFO] [main] |resources.MemoryDistributor|: InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=2291138560, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor 2019-04-16 11:51:50,060 [INFO] [TezChild] |task.TaskRunner2Callable|: Initializing task, taskAttemptId=attempt_1555406634668_0016_1_00_000000_0 2019-04-16 11:51:50,074 [INFO] [TaskHeartbeatThread] |task.TaskReporter|: Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1555406634668_0016_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0 2019-04-16 11:51:50,078 [INFO] [TezChild] |Configuration.deprecation|: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed 2019-04-16 11:51:50,079 [INFO] [TezChild] |Configuration.deprecation|: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive 2019-04-16 11:51:50,088 [INFO] [TezChild] |Configuration.deprecation|: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 2019-04-16 11:51:50,088 [INFO] [TezChild] |Configuration.deprecation|: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition 2019-04-16 11:51:50,088 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor 2019-04-16 11:51:50,088 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish 2019-04-16 11:51:50,099 [INFO] [I/O Setup 1 Initialize: {out_Map 1}] |counters.Limits|: Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200 2019-04-16 11:51:50,102 [INFO] [I/O Setup 0 Initialize: {values__tmp__table__1}] |input.MRInput|: values__tmp__table__1 using newmapreduce API=false, split via event=true, numPhysicalInputs=1 2019-04-16 11:51:50,102 [INFO] [I/O Setup 0 Initialize: {values__tmp__table__1}] |input.MRInputLegacy|: values__tmp__table__1 MRInputLegacy deferring initialization 2019-04-16 11:51:50,102 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish 2019-04-16 11:51:50,106 [INFO] [I/O Setup 1 Initialize: {out_Map 1}] |output.MROutput|: out_Map 1: outputFormat=org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl, using newmapreduce API=false 2019-04-16 11:51:50,532 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: All initializers finished 2019-04-16 11:51:50,534 [INFO] [TezChild] |resources.MemoryDistributor|: InitialRequests=[out_Map 1:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [values__tmp__table__1:INPUT:0:org.apache.tez.mapreduce.input.MRInputLegacy] 2019-04-16 11:51:50,569 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1] 2019-04-16 11:51:50,572 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996 2019-04-16 11:51:50,572 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: Scaling Requests. NumRequests: 2, numScaledRequests: 2, TotalRequested: 0, TotalRequestedScaled: 0.0, TotalJVMHeap: 2291138560, TotalAvailable: 1535062835, TotalRequested/TotalJVMHeap:0.00 2019-04-16 11:51:50,572 [INFO] [TezChild] |resources.MemoryDistributor|: Allocations=[out_Map 1:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [values__tmp__table__1:org.apache.tez.mapreduce.input.MRInputLegacy:INPUT:0:0] 2019-04-16 11:51:50,572 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Starting Inputs/Outputs 2019-04-16 11:51:50,573 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Input: values__tmp__table__1 being auto started by the framework. Subsequent instances will not be auto-started 2019-04-16 11:51:50,573 [INFO] [I/O Setup 0 Start: {values__tmp__table__1}] |runtime.LogicalIOProcessorRuntimeTask|: Started Input with src edge: values__tmp__table__1 2019-04-16 11:51:50,573 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Num IOs determined for AutoStart: 1 2019-04-16 11:51:50,573 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 IOs to start 2019-04-16 11:51:50,573 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: AutoStartComplete 2019-04-16 11:51:50,573 [INFO] [TezChild] |task.TaskRunner2Callable|: Running task, taskAttemptId=attempt_1555406634668_0016_1_00_000000_0 2019-04-16 11:51:50,631 [INFO] [TezChild] |conf.HiveConf|: Found configuration file null 2019-04-16 11:51:50,704 [INFO] [TezChild] |exec.Utilities|: PLAN PATH = hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/hive_2019-04-16_11-51-36_651_399717790387024222-2/hadoop/_tez_scratch_dir/c2b90fcf-48b3-498b-8ee8-ee021d9875b7/map.xml 2019-04-16 11:51:50,706 [INFO] [TezChild] |exec.SerializationUtilities|: Deserializing MapWork using kryo 2019-04-16 11:51:50,784 [INFO] [TezChild] |exec.Utilities|: Deserialized plan (via RPC) - name: Map 1 size: 2.19KB 2019-04-16 11:51:50,784 [INFO] [TezChild] |tez.ObjectCache|: Caching key: hive_20190416115136_f07651b2-c968-4408-9a66-eb636ae8a49a_Map 1__MAP_PLAN__ 2019-04-16 11:51:50,787 [INFO] [TezChild] |tez.MapRecordProcessor|: The input names are: [values__tmp__table__1] 2019-04-16 11:51:50,788 [INFO] [TezChild] |input.MRInput|: Initialized MRInput: values__tmp__table__1 2019-04-16 11:51:50,797 [INFO] [TezChild] |exec.Utilities|: PLAN PATH = hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/hive_2019-04-16_11-51-36_651_399717790387024222-2/hadoop/_tez_scratch_dir/c2b90fcf-48b3-498b-8ee8-ee021d9875b7/map.xml 2019-04-16 11:51:50,806 [INFO] [TezChild] |lzo.GPLNativeCodeLoader|: Loaded native gpl library 2019-04-16 11:51:50,808 [INFO] [TezChild] |lzo.LzoCodec|: Successfully loaded & initialized native-lzo library [hadoop-lzo rev bab859f34a291cb7b3f4e724b59e1b48af69016b] 2019-04-16 11:51:50,858 [INFO] [TezChild] |lib.MRReaderMapred|: Processing split: TezGroupedSplit{wrappedSplits=[org.apache.hadoop.mapred.TextInputFormat:hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/_tmp_space.db/Values__Tmp__Table__1/data_file:0+2], wrappedInputFormatName='org.apache.hadoop.hive.ql.io.HiveInputFormat', locations=[ip-172-31-92-1.ec2.internal], rack='null', length=2} 2019-04-16 11:51:50,858 [INFO] [TezChild] |input.MRInput|: values__tmp__table__1 initialized RecordReader from event 2019-04-16 11:51:50,860 [INFO] [TezChild] |tez.MapRecordProcessor|: Main input name is Map 1 2019-04-16 11:51:50,860 [INFO] [TezChild] |exec.MapOperator|: Initializing operator MAP[0] 2019-04-16 11:51:50,917 [INFO] [TezChild] |tez.MapRecordProcessor|: Id =0 null <\Children> <\Parent> <\MAP> 2019-04-16 11:51:50,918 [INFO] [TezChild] |tez.MapRecordProcessor|: Memory available for operators set to 2.13GB 2019-04-16 11:51:50,920 [INFO] [TezChild] |tez.ObjectCache|: Caching key: hive_20190416115136_f07651b2-c968-4408-9a66-eb636ae8a49a_DynamicValueRegistry 2019-04-16 11:51:50,920 [INFO] [TezChild] |exec.TableScanOperator|: Initializing operator TS[0] 2019-04-16 11:51:50,920 [INFO] [TezChild] |exec.SelectOperator|: Initializing operator SEL[1] 2019-04-16 11:51:51,061 [INFO] [TezChild] |exec.SelectOperator|: SELECT struct 2019-04-16 11:51:51,064 [INFO] [TezChild] |exec.FileSinkOperator|: Initializing operator FS[3] 2019-04-16 11:51:51,065 [INFO] [TezChild] |exec.FileSinkOperator|: Using serializer : class org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe[[[B@38f519b]:[rownum]:[int]] and formatter : org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat@2065d06 2019-04-16 11:51:51,065 [INFO] [TezChild] |Configuration.deprecation|: mapred.healthChecker.script.timeout is deprecated. Instead, use mapreduce.tasktracker.healthchecker.script.timeout 2019-04-16 11:51:51,067 [INFO] [TezChild] |exec.Utilities|: PLAN PATH = hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/hive_2019-04-16_11-51-36_651_399717790387024222-2/hadoop/_tez_scratch_dir/c2b90fcf-48b3-498b-8ee8-ee021d9875b7/map.xml 2019-04-16 11:51:51,121 [INFO] [TezChild] |exec.FileSinkOperator|: Final Path: FS hdfs://ip-172-31-90-92.ec2.internal:8020/user/hive/warehouse/idtable/.hive-staging_hive_2019-04-16_11-51-36_651_399717790387024222-1/_tmp.-ext-10000/000000_0 2019-04-16 11:51:51,121 [INFO] [TezChild] |exec.FileSinkOperator|: Writing to temp file: FS hdfs://ip-172-31-90-92.ec2.internal:8020/user/hive/warehouse/idtable/.hive-staging_hive_2019-04-16_11-51-36_651_399717790387024222-1/_task_tmp.-ext-10000/_tmp.000000_0 2019-04-16 11:51:51,121 [INFO] [TezChild] |exec.FileSinkOperator|: New Final Path: FS hdfs://ip-172-31-90-92.ec2.internal:8020/user/hive/warehouse/idtable/.hive-staging_hive_2019-04-16_11-51-36_651_399717790387024222-1/_tmp.-ext-10000/000000_0 2019-04-16 11:51:51,166 [INFO] [TezChild] |exec.FileSinkOperator|: FS[3]: records written - 1 2019-04-16 11:51:51,166 [INFO] [TezChild] |exec.MapOperator|: MAP[0]: records read - 1 2019-04-16 11:51:51,167 [INFO] [TezChild] |tez.ObjectCache|: Releasing key: hive_20190416115136_f07651b2-c968-4408-9a66-eb636ae8a49a_Map 1__MAP_PLAN__ 2019-04-16 11:51:51,167 [INFO] [TezChild] |tez.ObjectCache|: Releasing key: hive_20190416115136_f07651b2-c968-4408-9a66-eb636ae8a49a_DynamicValueRegistry 2019-04-16 11:51:51,167 [INFO] [TezChild] |exec.MapOperator|: Closing operator MAP[0] 2019-04-16 11:51:51,167 [INFO] [TezChild] |exec.MapOperator|: DESERIALIZE_ERRORS:0, RECORDS_IN_Map_1:1, 2019-04-16 11:51:51,167 [INFO] [TezChild] |exec.TableScanOperator|: Closing operator TS[0] 2019-04-16 11:51:51,167 [INFO] [TezChild] |exec.SelectOperator|: Closing operator SEL[1] 2019-04-16 11:51:51,167 [INFO] [TezChild] |exec.FileSinkOperator|: Closing operator FS[3] 2019-04-16 11:51:51,167 [INFO] [TezChild] |exec.FileSinkOperator|: FS[3]: records written - 1 2019-04-16 11:51:52,011 [INFO] [TezChild] |exec.FileSinkOperator|: RECORDS_OUT_1_default.idtable:1, 2019-04-16 11:51:52,011 [INFO] [TezChild] |task.TaskRunner2Callable|: Closing task, taskAttemptId=attempt_1555406634668_0016_1_00_000000_0 2019-04-16 11:51:52,011 [INFO] [TezChild] |output.MROutput|: out_Map 1 closed 2019-04-16 11:51:52,012 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Joining on EventRouter 2019-04-16 11:51:52,015 [INFO] [TezChild] |task.TaskRunner2Callable|: Task completed, taskAttemptId=attempt_1555406634668_0016_1_00_000000_0, askedToStop=false 2019-04-16 11:51:52,015 [INFO] [TezChild] |task.TaskRunner2Callable|: Cleaning up task attempt_1555406634668_0016_1_00_000000_0, stopRequested=false 2019-04-16 11:51:52,017 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1555406634668_0016_1_00_000000_0: Counters: 17 [[File System Counters HDFS_BYTES_READ=2, HDFS_BYTES_WRITTEN=73, HDFS_READ_OPS=4, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=3][org.apache.tez.common.counters.TaskCounter GC_TIME_MILLIS=72, CPU_MILLISECONDS=2910, PHYSICAL_MEMORY_BYTES=315621376, VIRTUAL_MEMORY_BYTES=4534431744, COMMITTED_HEAP_BYTES=315621376, INPUT_RECORDS_PROCESSED=1, INPUT_SPLIT_LENGTH_BYTES=2, OUTPUT_RECORDS=0][HIVE CREATED_FILES=1, DESERIALIZE_ERRORS=0, RECORDS_IN_Map_1=1, RECORDS_OUT_1_default.idtable=1]] 2019-04-16 11:51:52,025 [INFO] [main] |task.TezTaskRunner2|: TaskRunnerResult for attempt_1555406634668_0016_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, containerShutdownRequested=false} 2019-04-16 11:51:52,025 [INFO] [main] |task.TezChild|: TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, containerShutdownRequested=false} 2019-04-16 11:51:52,126 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: End of LogType:syslog_attempt_1555406634668_0016_1_00_000000_0 Container: container_1555406634668_0016_01_000003 on ip-172-31-91-60.ec2.internal_8041 ======================================================================================== LogType:stderr Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:188 Log Contents: 2019-04-16 11:53:12 Starting to run new task attempt: attempt_1555406634668_0016_2_00_000000_0 2019-04-16 11:53:19 Completed running task attempt: attempt_1555406634668_0016_2_00_000000_0 End of LogType:stderr LogType:stdout Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:188 Log Contents: 2019-04-16 11:53:12 Starting to run new task attempt: attempt_1555406634668_0016_2_00_000000_0 2019-04-16 11:53:19 Completed running task attempt: attempt_1555406634668_0016_2_00_000000_0 End of LogType:stdout LogType:syslog Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:39366 Log Contents: 2019-04-16 11:53:11,910 [INFO] [main] |task.TezChild|: TezChild starting with PID=17345, containerIdentifier=container_1555406634668_0016_01_000003 2019-04-16 11:53:12,366 [INFO] [main] |shim.HadoopShimsLoader|: Trying to locate HadoopShimProvider for hadoopVersion=2.8.5-amzn-1, majorVersion=2, minorVersion=8 2019-04-16 11:53:12,367 [INFO] [main] |shim.HadoopShimsLoader|: Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.8.5-amzn-1, majorVersion=2, minorVersion=8 2019-04-16 11:53:12,394 [INFO] [main] |task.TezChild|: /************************************************************ [system properties] os.name: Linux os.version: 4.14.77-70.82.amzn1.x86_64 java.home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.42.amzn1.x86_64/jre java.runtime.version: 1.8.0_191-b12 java.vendor: Oracle Corporation java.version: 1.8.0_191 java.vm.name: OpenJDK 64-Bit Server VM java.class.path: /mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/emr-ddb-hive.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/emr-kinesis-hive.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/hive-openx-serde.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/emr-hive-goodies.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/hive-hcatalog-server-extensions-2.3.4-amzn-0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/emr-s3-select-hive-connector.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/TimestampUDF-1.0-SNAPSHOT.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/hive-exec-2.3.4-amzn-0-b4b465141bab0a22bba33ac51550e7bd41a2a42cf3dd3f440846c25eb6e4c431.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/hive-hcatalog-pig-adapter-2.3.4-amzn-0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/hive-hcatalog-core-2.3.4-amzn-0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/aws-glue-datacatalog-hive2-client.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/hive-hcatalog-streaming-2.3.4-amzn-0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/tez-yarn-timeline-history-with-acls-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/tez-tests-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/tez-mapreduce-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/tez-ext-service-tests-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/tez-runtime-library-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/tez-dag-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/hadoop-shim-2.7-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/tez-history-parser-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/tez-runtime-internals-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/tez-job-analyzer-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/tez-examples-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/hadoop-shim-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/tez-common-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/tez-javadoc-tools-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/tez-yarn-timeline-history-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/tez-api-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/commons-cli-1.2.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/hadoop-mapreduce-client-core-2.8.5-amzn-1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/commons-math3-3.1.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/commons-collections-3.2.2.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/guava-11.0.2.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/commons-lang-2.6.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/slf4j-api-1.7.10.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/netty-3.6.2.Final.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/commons-codec-1.4.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/commons-io-2.4.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/async-http-client-1.8.16.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/RoaringBitmap-0.4.9.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/jettison-1.3.4.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/jetty-util-6.1.26.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/jsr305-3.0.0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/jetty-6.1.26.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/servlet-api-2.5.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/jersey-client-1.9.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/metrics-core-3.1.0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/commons-collections4-4.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/protobuf-java-2.5.0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/hadoop-mapreduce-client-common-2.8.5-amzn-1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/jersey-json-1.9.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/hadoop-mapreduce-client-common.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/hhadoop-mapreduce-client-core.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/hadoop-auth.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/hadoop-annotations.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tezlib/lib/hadoop-yarn-server-web-proxy.jar:/etc/hadoop/conf:/usr/lib/hadoop/hadoop-rumen-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-distcp-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-azure-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/hadoop-streaming-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-openstack-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/hadoop/hadoop-openstack.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-sls-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-extras-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-datajoin-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-auth-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-archives.jar:/usr/lib/hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/hadoop-annotations-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/hadoop/hadoop-common-2.8.5-amzn-1-tests.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-gridmix-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-ant-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-nfs-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-aws-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-streaming.jar:/usr/lib/hadoop/hadoop-sls.jar:/usr/lib/hadoop/hadoop-common-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-azure-datalake.jar:/usr/lib/hadoop/hadoop-archives-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.5-amzn-1.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/nimbus-jose-jwt-4.41.1.jar:/usr/lib/hadoop/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/audience-annotations-0.5.0.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/jcip-annotations-1.0-1.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/json-smart-1.3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/httpcore-4.4.9.jar:/usr/lib/hadoop/lib/zookeeper-3.4.13.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/httpclient-4.5.5.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/jsch-0.1.54.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.5-amzn-1-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.5-amzn-1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.5-amzn-1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.5-amzn-1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.8.5-amzn-1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.5-amzn-1-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.5-amzn-1-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/okio-1.4.0.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/okhttp-2.4.0.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/nimbus-jose-jwt-4.41.1.jar:/usr/lib/hadoop-mapreduce/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/azure-data-lake-store-sdk-2.2.3.jar:/usr/lib/hadoop-mapreduce/commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/audience-annotations-0.5.0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/jcip-annotations-1.0-1.jar:/usr/lib/hadoop-mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-core-1.11.461.jar:/usr/lib/hadoop-mapreduce/json-smart-1.3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/httpcore-4.4.9.jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.7.jar:/usr/lib/hadoop-mapreduce/zookeeper-3.4.13.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-extras-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-auth-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-archives.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/httpclient-4.5.5.jar:/usr/lib/hadoop-mapreduce/joda-time-2.9.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/okio-1.4.0.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.54.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/azure-storage-2.2.0.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.5-amzn-1-tests.jar:/usr/lib/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/jackson-databind-2.6.7.2.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/okhttp-2.4.0.jar:/usr/lib/hadoop-mapreduce/jmespath-java-1.11.461.jar:/usr/lib/hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/hadoop-ant-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/jackson-dataformat-cbor-2.6.7.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/jackson-annotations-2.6.7.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs.jar:/usr/lib/hadoop-mapreduce/hadoop-aws-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-s3-1.11.461.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-kms-1.11.461.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/lib/curator-test-2.7.1.jar:/usr/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.13-tests.jar:/usr/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/lib/hadoop-yarn/lib/audience-annotations-0.5.0.jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.13.jar:/usr/lib/hadoop-yarn/lib/commons-math-2.2.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/lib/hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/lib/hadoop-yarn/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.11.393.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/jcl-over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.29.0.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar:/usr/lib/spark/yarn/lib/datanucleus-core.jar:/usr/lib/spark/yarn/lib/datanucleus-rdbms.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.1.0.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticsearch-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-storagegateway-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-clouddirectory-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacemeteringservice-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-signer-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-snowball-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticbeanstalk-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitosync-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloud9-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3control-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastoredata-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscalingplans-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediapackage-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-secretsmanager-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-transcribe-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-applicationautoscaling-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicediscovery-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediaconvert-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-licensemanager-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplaceentitlement-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-translate-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticache-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-transfer-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-migrationhub-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidentity-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-fms-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-machinelearning-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directory-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elastictranscoder-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpleworkflow-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacecommerceanalytics-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastore-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-xray-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dynamodb-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-connect-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-macie-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lexmodelbuilding-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpointemail-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesisvideo-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ram-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicecatalog-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-serverlessapplicationrepository-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidp-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesisanalyticsv2-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-support-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mq-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-comprehendmedical-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecr-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-waf-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediatailor-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-eks-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-robomaker-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workmail-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costandusagereport-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-stepfunctions-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-guardduty-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-discovery-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpoint-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mobile-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workdocs-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glue-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-code-generator-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-medialive-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot1clickdevices-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-chime-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-organizations-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemakerruntime-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dlm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudformation-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rekognition-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot1clickprojects-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-health-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appmesh-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworkscm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sns-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-greengrass-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servermigration-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-fsx-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-logs-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api-gateway-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-alexaforbusiness-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pi-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-neptune-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-batch-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudtrail-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-amplify-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glacier-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-datasync-1.11.461.jar:/usr/share/aws/aws-java-sdk/jmespath-java-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-quicksight-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-budgets-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemaker-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rdsdata-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dax-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-globalaccelerator-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancing-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroups-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kafka-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecs-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iotanalytics-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpointsmsvoice-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iotjobsdataplane-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworks-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsmv2-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesis-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-datapipeline-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pricing-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lightsail-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediaconnect-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-comprehend-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-devicefarm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpledb-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudfront-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acmpca-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53resolver-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancingv2-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codepipeline-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-events-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codestar-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatch-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costexplorer-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appsync-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iam-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-securityhub-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kms-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11.461.jar: java.io.tmpdir: /mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003/tmp user.dir: /mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000003 user.name: yarn ************************************************************/ 2019-04-16 11:53:12,436 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_1555406634668_0016_01_000003 2019-04-16 11:53:12,574 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_1555406634668_0016_01_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1555406634668_0016_2_00_000000_0 2019-04-16 11:53:12,576 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: attempt_1555406634668_0016_2_00_000000_0 2019-04-16 11:53:19,860 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_1555406634668_0016_01_000003 End of LogType:syslog LogType:syslog_attempt_1555406634668_0016_2_00_000000_0 Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:14068 Log Contents: 2019-04-16 11:53:12,583 [INFO] [main] |task.TezChild|: Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0 2019-04-16 11:53:12,602 [INFO] [main] |metrics.TaskCounterUpdater|: Using ResourceCalculatorProcessTree : org.apache.tez.util.TezMxBeanResourceCalculator 2019-04-16 11:53:12,602 [INFO] [main] |runtime.LogicalIOProcessorRuntimeTask|: Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : Select T.Candidate.startTime, T.Candidat...T(Stage-1), VertexName: Map 1, VertexParallelism: 1, TaskAttemptID:attempt_1555406634668_0016_2_00_000000_0, processorName=org.apache.hadoop.hive.ql.exec.tez.MapTezProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=idtable, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInputLegacy }}, ], outputSpecList=[{{ destinationVertexName=out_Map 1, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ], taskConfEntryCount=0 2019-04-16 11:53:12,604 [INFO] [main] |resources.MemoryDistributor|: InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=2291138560, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor 2019-04-16 11:53:12,613 [INFO] [TezChild] |task.TaskRunner2Callable|: Initializing task, taskAttemptId=attempt_1555406634668_0016_2_00_000000_0 2019-04-16 11:53:12,624 [INFO] [TaskHeartbeatThread] |task.TaskReporter|: Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1555406634668_0016_2_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0 2019-04-16 11:53:12,632 [INFO] [TezChild] |Configuration.deprecation|: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed 2019-04-16 11:53:12,633 [INFO] [TezChild] |Configuration.deprecation|: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive 2019-04-16 11:53:12,639 [INFO] [TezChild] |Configuration.deprecation|: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 2019-04-16 11:53:12,639 [INFO] [TezChild] |Configuration.deprecation|: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition 2019-04-16 11:53:12,639 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor 2019-04-16 11:53:12,639 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish 2019-04-16 11:53:12,649 [INFO] [I/O Setup 1 Initialize: {out_Map 1}] |counters.Limits|: Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200 2019-04-16 11:53:12,655 [INFO] [I/O Setup 1 Initialize: {out_Map 1}] |output.MROutput|: out_Map 1: outputFormat=org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl, using newmapreduce API=false 2019-04-16 11:53:12,656 [INFO] [I/O Setup 0 Initialize: {idtable}] |input.MRInput|: idtable using newmapreduce API=false, split via event=true, numPhysicalInputs=1 2019-04-16 11:53:12,656 [INFO] [I/O Setup 0 Initialize: {idtable}] |input.MRInputLegacy|: idtable MRInputLegacy deferring initialization 2019-04-16 11:53:12,656 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish 2019-04-16 11:53:13,043 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: All initializers finished 2019-04-16 11:53:13,045 [INFO] [TezChild] |resources.MemoryDistributor|: InitialRequests=[out_Map 1:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [idtable:INPUT:0:org.apache.tez.mapreduce.input.MRInputLegacy] 2019-04-16 11:53:13,081 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1] 2019-04-16 11:53:13,084 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996 2019-04-16 11:53:13,084 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: Scaling Requests. NumRequests: 2, numScaledRequests: 2, TotalRequested: 0, TotalRequestedScaled: 0.0, TotalJVMHeap: 2291138560, TotalAvailable: 1535062835, TotalRequested/TotalJVMHeap:0.00 2019-04-16 11:53:13,084 [INFO] [TezChild] |resources.MemoryDistributor|: Allocations=[out_Map 1:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [idtable:org.apache.tez.mapreduce.input.MRInputLegacy:INPUT:0:0] 2019-04-16 11:53:13,084 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Starting Inputs/Outputs 2019-04-16 11:53:13,085 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Input: idtable being auto started by the framework. Subsequent instances will not be auto-started 2019-04-16 11:53:13,085 [INFO] [I/O Setup 0 Start: {idtable}] |runtime.LogicalIOProcessorRuntimeTask|: Started Input with src edge: idtable 2019-04-16 11:53:13,085 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Num IOs determined for AutoStart: 1 2019-04-16 11:53:13,085 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 IOs to start 2019-04-16 11:53:13,085 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: AutoStartComplete 2019-04-16 11:53:13,086 [INFO] [TezChild] |task.TaskRunner2Callable|: Running task, taskAttemptId=attempt_1555406634668_0016_2_00_000000_0 2019-04-16 11:53:13,134 [INFO] [TezChild] |conf.HiveConf|: Found configuration file null 2019-04-16 11:53:13,206 [INFO] [TezChild] |exec.Utilities|: PLAN PATH = hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/hive_2019-04-16_11-52-50_757_5369209958768659300-3/hadoop/_tez_scratch_dir/35664f3e-1811-4541-ba57-f45dad3372f5/map.xml 2019-04-16 11:53:13,207 [INFO] [TezChild] |exec.SerializationUtilities|: Deserializing MapWork using kryo 2019-04-16 11:53:13,290 [INFO] [TezChild] |exec.Utilities|: Deserialized plan (via RPC) - name: Map 1 size: 2.33KB 2019-04-16 11:53:13,290 [INFO] [TezChild] |tez.ObjectCache|: Caching key: hive_20190416115250_eab01a63-574a-4628-a799-bd359db0122c_Map 1__MAP_PLAN__ 2019-04-16 11:53:13,293 [INFO] [TezChild] |tez.MapRecordProcessor|: The input names are: [idtable] 2019-04-16 11:53:13,294 [INFO] [TezChild] |input.MRInput|: Initialized MRInput: idtable 2019-04-16 11:53:13,307 [INFO] [TezChild] |exec.Utilities|: PLAN PATH = hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/hive_2019-04-16_11-52-50_757_5369209958768659300-3/hadoop/_tez_scratch_dir/35664f3e-1811-4541-ba57-f45dad3372f5/map.xml 2019-04-16 11:53:13,330 [INFO] [TezChild] |lzo.GPLNativeCodeLoader|: Loaded native gpl library 2019-04-16 11:53:13,332 [INFO] [TezChild] |lzo.LzoCodec|: Successfully loaded & initialized native-lzo library [hadoop-lzo rev bab859f34a291cb7b3f4e724b59e1b48af69016b] 2019-04-16 11:53:13,386 [INFO] [TezChild] |lib.MRReaderMapred|: Processing split: TezGroupedSplit{wrappedSplits=[org.apache.hadoop.mapred.TextInputFormat:hdfs://ip-172-31-90-92.ec2.internal:8020/user/hive/warehouse/idtable/000000_0:0+2], wrappedInputFormatName='org.apache.hadoop.hive.ql.io.HiveInputFormat', locations=[ip-172-31-91-60.ec2.internal], rack='null', length=2} 2019-04-16 11:53:13,387 [INFO] [TezChild] |input.MRInput|: idtable initialized RecordReader from event 2019-04-16 11:53:13,389 [INFO] [TezChild] |tez.MapRecordProcessor|: Main input name is Map 1 2019-04-16 11:53:13,389 [INFO] [TezChild] |exec.MapOperator|: Initializing operator MAP[0] 2019-04-16 11:53:13,439 [INFO] [TezChild] |tez.MapRecordProcessor|: Id =0 null <\Children> <\Parent> <\MAP> 2019-04-16 11:53:13,440 [INFO] [TezChild] |tez.MapRecordProcessor|: Memory available for operators set to 2.13GB 2019-04-16 11:53:13,441 [INFO] [TezChild] |tez.ObjectCache|: Caching key: hive_20190416115250_eab01a63-574a-4628-a799-bd359db0122c_DynamicValueRegistry 2019-04-16 11:53:13,442 [INFO] [TezChild] |exec.TableScanOperator|: Initializing operator TS[0] 2019-04-16 11:53:13,442 [INFO] [TezChild] |exec.SelectOperator|: Initializing operator SEL[1] 2019-04-16 11:53:13,446 [INFO] [TezChild] |exec.SelectOperator|: SELECT struct 2019-04-16 11:53:13,594 [INFO] [TezChild] |hive.TimestampUDF|: == TimestampUDF == 2019-04-16 11:53:13,616 [INFO] [TezChild] |hive.TimestampUDF|: startTime: 2019-04-16T11:53:13.595Z 2019-04-16 11:53:15,616 [INFO] [TezChild] |hive.TimestampUDF|: endTime: 2019-04-16T11:53:15.616Z 2019-04-16 11:53:15,616 [INFO] [TezChild] |hive.TimestampUDF|: duration: 2021 2019-04-16 11:53:15,618 [INFO] [TezChild] |hive.TimestampUDF|: == TimestampUDF == 2019-04-16 11:53:15,618 [INFO] [TezChild] |hive.TimestampUDF|: startTime: 2019-04-16T11:53:15.618Z 2019-04-16 11:53:17,619 [INFO] [TezChild] |hive.TimestampUDF|: endTime: 2019-04-16T11:53:17.619Z 2019-04-16 11:53:17,619 [INFO] [TezChild] |hive.TimestampUDF|: duration: 2001 2019-04-16 11:53:17,619 [INFO] [TezChild] |hive.TimestampUDF|: == TimestampUDF == 2019-04-16 11:53:17,619 [INFO] [TezChild] |hive.TimestampUDF|: startTime: 2019-04-16T11:53:17.619Z 2019-04-16 11:53:19,619 [INFO] [TezChild] |hive.TimestampUDF|: endTime: 2019-04-16T11:53:19.619Z 2019-04-16 11:53:19,619 [INFO] [TezChild] |hive.TimestampUDF|: duration: 2000 2019-04-16 11:53:19,620 [INFO] [TezChild] |exec.FileSinkOperator|: Initializing operator FS[2] 2019-04-16 11:53:19,621 [INFO] [TezChild] |exec.FileSinkOperator|: Using serializer : class org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe[[[B@47607d42]:[_col0, _col1, _col2]:[string, string, bigint]] and formatter : org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat@6d348304 2019-04-16 11:53:19,621 [INFO] [TezChild] |Configuration.deprecation|: mapred.healthChecker.script.timeout is deprecated. Instead, use mapreduce.tasktracker.healthchecker.script.timeout 2019-04-16 11:53:19,623 [INFO] [TezChild] |exec.Utilities|: PLAN PATH = hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/hive_2019-04-16_11-52-50_757_5369209958768659300-3/hadoop/_tez_scratch_dir/35664f3e-1811-4541-ba57-f45dad3372f5/map.xml 2019-04-16 11:53:19,674 [INFO] [TezChild] |exec.FileSinkOperator|: Final Path: FS hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/hive_2019-04-16_11-52-50_757_5369209958768659300-1/-mr-10001/.hive-staging_hive_2019-04-16_11-52-50_757_5369209958768659300-1/_tmp.-ext-10002/000000_0 2019-04-16 11:53:19,674 [INFO] [TezChild] |exec.FileSinkOperator|: Writing to temp file: FS hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/hive_2019-04-16_11-52-50_757_5369209958768659300-1/-mr-10001/.hive-staging_hive_2019-04-16_11-52-50_757_5369209958768659300-1/_task_tmp.-ext-10002/_tmp.000000_0 2019-04-16 11:53:19,674 [INFO] [TezChild] |exec.FileSinkOperator|: New Final Path: FS hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/hive_2019-04-16_11-52-50_757_5369209958768659300-1/-mr-10001/.hive-staging_hive_2019-04-16_11-52-50_757_5369209958768659300-1/_tmp.-ext-10002/000000_0 2019-04-16 11:53:19,724 [INFO] [TezChild] |exec.FileSinkOperator|: FS[2]: records written - 1 2019-04-16 11:53:19,724 [INFO] [TezChild] |exec.MapOperator|: MAP[0]: records read - 1 2019-04-16 11:53:19,725 [INFO] [TezChild] |tez.ObjectCache|: Releasing key: hive_20190416115250_eab01a63-574a-4628-a799-bd359db0122c_Map 1__MAP_PLAN__ 2019-04-16 11:53:19,725 [INFO] [TezChild] |tez.ObjectCache|: Releasing key: hive_20190416115250_eab01a63-574a-4628-a799-bd359db0122c_DynamicValueRegistry 2019-04-16 11:53:19,725 [INFO] [TezChild] |exec.MapOperator|: Closing operator MAP[0] 2019-04-16 11:53:19,725 [INFO] [TezChild] |exec.MapOperator|: DESERIALIZE_ERRORS:0, RECORDS_IN_Map_1:1, 2019-04-16 11:53:19,725 [INFO] [TezChild] |exec.TableScanOperator|: Closing operator TS[0] 2019-04-16 11:53:19,725 [INFO] [TezChild] |exec.SelectOperator|: Closing operator SEL[1] 2019-04-16 11:53:19,725 [INFO] [TezChild] |exec.FileSinkOperator|: Closing operator FS[2] 2019-04-16 11:53:19,725 [INFO] [TezChild] |exec.FileSinkOperator|: FS[2]: records written - 1 2019-04-16 11:53:19,751 [INFO] [TezChild] |exec.FileSinkOperator|: RECORDS_OUT_0:1, 2019-04-16 11:53:19,751 [INFO] [TezChild] |task.TaskRunner2Callable|: Closing task, taskAttemptId=attempt_1555406634668_0016_2_00_000000_0 2019-04-16 11:53:19,751 [INFO] [TezChild] |output.MROutput|: out_Map 1 closed 2019-04-16 11:53:19,751 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Joining on EventRouter 2019-04-16 11:53:19,754 [INFO] [TezChild] |task.TaskRunner2Callable|: Task completed, taskAttemptId=attempt_1555406634668_0016_2_00_000000_0, askedToStop=false 2019-04-16 11:53:19,754 [INFO] [TezChild] |task.TaskRunner2Callable|: Cleaning up task attempt_1555406634668_0016_2_00_000000_0, stopRequested=false 2019-04-16 11:53:19,755 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1555406634668_0016_2_00_000000_0: Counters: 17 [[File System Counters HDFS_BYTES_READ=2, HDFS_BYTES_WRITTEN=154, HDFS_READ_OPS=3, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=2][org.apache.tez.common.counters.TaskCounter GC_TIME_MILLIS=65, CPU_MILLISECONDS=2900, PHYSICAL_MEMORY_BYTES=308281344, VIRTUAL_MEMORY_BYTES=4532211712, COMMITTED_HEAP_BYTES=308281344, INPUT_RECORDS_PROCESSED=1, INPUT_SPLIT_LENGTH_BYTES=2, OUTPUT_RECORDS=0][HIVE CREATED_FILES=1, DESERIALIZE_ERRORS=0, RECORDS_IN_Map_1=1, RECORDS_OUT_0=1]] 2019-04-16 11:53:19,759 [INFO] [main] |task.TezTaskRunner2|: TaskRunnerResult for attempt_1555406634668_0016_2_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, containerShutdownRequested=false} 2019-04-16 11:53:19,759 [INFO] [main] |task.TezChild|: TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, containerShutdownRequested=false} 2019-04-16 11:53:19,860 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: End of LogType:syslog_attempt_1555406634668_0016_2_00_000000_0 Container: container_1555406634668_0016_01_000001 on ip-172-31-92-1.ec2.internal_8041 ======================================================================================= LogType:dag_1555406634668_0016_1.dot Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:1055 Log Contents: digraph insert_overwrite_table_idTable_values_1_Stage_1_ { graph [ label="insert_overwrite_table_idTable_values_1_Stage_1_", fontsize=24, fontname=Helvetica]; node [fontsize=12, fontname=Helvetica]; edge [fontsize=9, fontcolor=blue, fontname=Arial]; "insert_overwrite_table_idTable_values_1_Stage_1_.Map_1" [ label = "Map_1[MapTezProcessor]" , color= "black" ]; "insert_overwrite_table_idTable_values_1_Stage_1_.Map_1" -> "insert_overwrite_table_idTable_values_1_Stage_1_.Map_1_out_Map_1" [ label = "Output [outputClass=MROutput,\n committer=]" ]; "insert_overwrite_table_idTable_values_1_Stage_1_.Map_1_out_Map_1" [ label = "Map_1[out_Map_1]", shape = "box" , color= "black"]; "insert_overwrite_table_idTable_values_1_Stage_1_.Map_1_values__tmp__table__1" [ label = "Map_1[values__tmp__table__1]", shape = "box" , color= "black"]; "insert_overwrite_table_idTable_values_1_Stage_1_.Map_1_values__tmp__table__1" -> "insert_overwrite_table_idTable_values_1_Stage_1_.Map_1" [ label = "Input [inputClass=MRInputLegacy,\n initializer=HiveSplitGenerator]" ]; } End of LogType:dag_1555406634668_0016_1.dot LogType:dag_1555406634668_0016_2.dot Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:1031 Log Contents: digraph Select_T_Candidate_startTime_T_Candidat_T_Stage_1_ { graph [ label="Select_T_Candidate_startTime_T_Candidat_T_Stage_1_", fontsize=24, fontname=Helvetica]; node [fontsize=12, fontname=Helvetica]; edge [fontsize=9, fontcolor=blue, fontname=Arial]; "Select_T_Candidate_startTime_T_Candidat_T_Stage_1_.Map_1_out_Map_1" [ label = "Map_1[out_Map_1]", shape = "box" , color= "black"]; "Select_T_Candidate_startTime_T_Candidat_T_Stage_1_.Map_1_idtable" [ label = "Map_1[idtable]", shape = "box" , color= "black"]; "Select_T_Candidate_startTime_T_Candidat_T_Stage_1_.Map_1_idtable" -> "Select_T_Candidate_startTime_T_Candidat_T_Stage_1_.Map_1" [ label = "Input [inputClass=MRInputLegacy,\n initializer=HiveSplitGenerator]" ]; "Select_T_Candidate_startTime_T_Candidat_T_Stage_1_.Map_1" [ label = "Map_1[MapTezProcessor]" , color= "black" ]; "Select_T_Candidate_startTime_T_Candidat_T_Stage_1_.Map_1" -> "Select_T_Candidate_startTime_T_Candidat_T_Stage_1_.Map_1_out_Map_1" [ label = "Output [outputClass=MROutput,\n committer=]" ]; } End of LogType:dag_1555406634668_0016_2.dot LogType:stderr Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:236 Log Contents: 2019-04-16 11:51:46 Running Dag: dag_1555406634668_0016_1 2019-04-16 11:51:52 Completed Dag: dag_1555406634668_0016_1 2019-04-16 11:53:09 Running Dag: dag_1555406634668_0016_2 2019-04-16 11:53:19 Completed Dag: dag_1555406634668_0016_2 End of LogType:stderr LogType:stdout Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:2688 Log Contents: 0.814: [GC (Allocation Failure) [PSYoungGen: 62464K->7378K(72704K)] 62464K->7466K(239616K), 0.0074495 secs] [Times: user=0.02 sys=0.01, real=0.00 secs] 1.339: [GC (Allocation Failure) [PSYoungGen: 69842K->9267K(72704K)] 69930K->9363K(239616K), 0.0097669 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 1.545: [GC (Allocation Failure) [PSYoungGen: 71731K->9129K(72704K)] 71827K->9233K(239616K), 0.0082490 secs] [Times: user=0.02 sys=0.01, real=0.01 secs] 1.617: [GC (Metadata GC Threshold) [PSYoungGen: 23005K->5992K(135168K)] 23109K->6104K(302080K), 0.0063920 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 1.623: [Full GC (Metadata GC Threshold) [PSYoungGen: 5992K->0K(135168K)] [ParOldGen: 112K->5940K(87552K)] 6104K->5940K(222720K), [Metaspace: 20916K->20916K(1069056K)], 0.0265886 secs] [Times: user=0.05 sys=0.02, real=0.03 secs] 2.251: [GC (Allocation Failure) [PSYoungGen: 124928K->8581K(135168K)] 130868K->14529K(222720K), 0.0097109 secs] [Times: user=0.02 sys=0.01, real=0.01 secs] 2.752: [GC (Allocation Failure) [PSYoungGen: 133509K->9997K(215040K)] 139457K->15954K(302592K), 0.0092913 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] 3.503: [GC (Metadata GC Threshold) [PSYoungGen: 163098K->11248K(218112K)] 169054K->17747K(305664K), 0.0183556 secs] [Times: user=0.05 sys=0.02, real=0.02 secs] 3.521: [Full GC (Metadata GC Threshold) [PSYoungGen: 11248K->0K(218112K)] [ParOldGen: 6498K->16562K(144384K)] 17747K->16562K(362496K), [Metaspace: 34883K->34883K(1081344K)], 0.1080242 secs] [Times: user=0.39 sys=0.01, real=0.11 secs] 2019-04-16 11:51:46 Running Dag: dag_1555406634668_0016_1 2019-04-16 11:51:52 Completed Dag: dag_1555406634668_0016_1 2019-04-16 11:53:09 Running Dag: dag_1555406634668_0016_2 2019-04-16 11:53:19 Completed Dag: dag_1555406634668_0016_2 242.345: [GC (Allocation Failure) [PSYoungGen: 206848K->12772K(259584K)] 223410K->34268K(403968K), 0.0152511 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] Heap PSYoungGen total 259584K, used 52502K [0x00000000eef00000, 0x0000000100000000, 0x0000000100000000) eden space 246784K, 16% used [0x00000000eef00000,0x00000000f15ccaf0,0x00000000fe000000) from space 12800K, 99% used [0x00000000ff380000,0x00000000ffff9058,0x0000000100000000) to space 16384K, 0% used [0x00000000fe000000,0x00000000fe000000,0x00000000ff000000) ParOldGen total 144384K, used 21496K [0x00000000ccc00000, 0x00000000d5900000, 0x00000000eef00000) object space 144384K, 14% used [0x00000000ccc00000,0x00000000ce0fe110,0x00000000d5900000) Metaspace used 48122K, capacity 48606K, committed 48896K, reserved 1093632K class space used 5382K, capacity 5539K, committed 5632K, reserved 1048576K End of LogType:stdout LogType:syslog Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:52122 Log Contents: 2019-04-16 11:51:43,741 [INFO] [main] |app.DAGAppMaster|: Creating DAGAppMaster for applicationId=application_1555406634668_0016, attemptNum=1, AMContainerId=container_1555406634668_0016_01_000001, jvmPid=17080, userFromEnv=hadoop, cliSessionOption=true, pwd=/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001, localDirs=/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016, logDirs=/var/log/hadoop-yarn/containers/application_1555406634668_0016/container_1555406634668_0016_01_000001 2019-04-16 11:51:44,145 [INFO] [main] |app.DAGAppMaster|: Created DAGAppMaster for application appattempt_1555406634668_0016_000001, versionInfo=[ component=tez-dag, version=0.9.1, revision=bab859f34a291cb7b3f4e724b59e1b48af69016b, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=2018-12-07T19:34:13Z ] 2019-04-16 11:51:44,178 [INFO] [main] |app.DAGAppMaster|: /************************************************************ [system properties] os.name: Linux os.version: 4.14.77-70.82.amzn1.x86_64 java.home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.42.amzn1.x86_64/jre java.runtime.version: 1.8.0_191-b12 java.vendor: Oracle Corporation java.version: 1.8.0_191 java.vm.name: OpenJDK 64-Bit Server VM java.class.path: /mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/emr-ddb-hive.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/emr-kinesis-hive.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/hive-openx-serde.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/emr-hive-goodies.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/hive-hcatalog-server-extensions-2.3.4-amzn-0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/emr-s3-select-hive-connector.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/hive-exec-2.3.4-amzn-0-b4b465141bab0a22bba33ac51550e7bd41a2a42cf3dd3f440846c25eb6e4c431.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/hive-hcatalog-pig-adapter-2.3.4-amzn-0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/hive-hcatalog-core-2.3.4-amzn-0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/aws-glue-datacatalog-hive2-client.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/hive-hcatalog-streaming-2.3.4-amzn-0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/tez-yarn-timeline-history-with-acls-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/tez-tests-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/tez-mapreduce-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/tez-ext-service-tests-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/tez-runtime-library-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/tez-dag-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/hadoop-shim-2.7-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/tez-history-parser-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/tez-runtime-internals-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/tez-job-analyzer-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/tez-examples-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/hadoop-shim-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/tez-common-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/tez-javadoc-tools-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/tez-yarn-timeline-history-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/tez-api-0.9.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/commons-cli-1.2.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/hadoop-mapreduce-client-core-2.8.5-amzn-1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/commons-math3-3.1.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/commons-collections-3.2.2.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/guava-11.0.2.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/commons-lang-2.6.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/slf4j-api-1.7.10.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/netty-3.6.2.Final.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/commons-codec-1.4.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/commons-io-2.4.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/async-http-client-1.8.16.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/RoaringBitmap-0.4.9.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/jettison-1.3.4.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/jetty-util-6.1.26.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/jsr305-3.0.0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/jetty-6.1.26.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/servlet-api-2.5.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/jersey-client-1.9.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/metrics-core-3.1.0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/commons-collections4-4.1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/protobuf-java-2.5.0.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/hadoop-mapreduce-client-common-2.8.5-amzn-1.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/jersey-json-1.9.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/hadoop-mapreduce-client-common.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/hhadoop-mapreduce-client-core.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/hadoop-auth.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/hadoop-annotations.jar:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tezlib/lib/hadoop-yarn-server-web-proxy.jar:/etc/hadoop/conf:/usr/lib/hadoop/hadoop-rumen-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-distcp-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-azure-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/hadoop-streaming-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-openstack-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/hadoop/hadoop-openstack.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-sls-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-extras-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-datajoin-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-auth-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-archives.jar:/usr/lib/hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/hadoop-annotations-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/hadoop/hadoop-common-2.8.5-amzn-1-tests.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-gridmix-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-ant-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-nfs-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-aws-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-streaming.jar:/usr/lib/hadoop/hadoop-sls.jar:/usr/lib/hadoop/hadoop-common-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-azure-datalake.jar:/usr/lib/hadoop/hadoop-archives-2.8.5-amzn-1.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.5-amzn-1.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/nimbus-jose-jwt-4.41.1.jar:/usr/lib/hadoop/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/audience-annotations-0.5.0.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/jcip-annotations-1.0-1.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/json-smart-1.3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/httpcore-4.4.9.jar:/usr/lib/hadoop/lib/zookeeper-3.4.13.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/httpclient-4.5.5.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/jsch-0.1.54.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.5-amzn-1-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.5-amzn-1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.5-amzn-1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.5-amzn-1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.8.5-amzn-1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.5-amzn-1-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.5-amzn-1-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/okio-1.4.0.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/okhttp-2.4.0.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/nimbus-jose-jwt-4.41.1.jar:/usr/lib/hadoop-mapreduce/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/azure-data-lake-store-sdk-2.2.3.jar:/usr/lib/hadoop-mapreduce/commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/audience-annotations-0.5.0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/jcip-annotations-1.0-1.jar:/usr/lib/hadoop-mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-core-1.11.461.jar:/usr/lib/hadoop-mapreduce/json-smart-1.3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/httpcore-4.4.9.jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.7.jar:/usr/lib/hadoop-mapreduce/zookeeper-3.4.13.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-extras-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-auth-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-archives.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/httpclient-4.5.5.jar:/usr/lib/hadoop-mapreduce/joda-time-2.9.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/okio-1.4.0.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.54.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/azure-storage-2.2.0.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.5-amzn-1-tests.jar:/usr/lib/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/jackson-databind-2.6.7.2.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/okhttp-2.4.0.jar:/usr/lib/hadoop-mapreduce/jmespath-java-1.11.461.jar:/usr/lib/hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/hadoop-ant-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/jackson-dataformat-cbor-2.6.7.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/jackson-annotations-2.6.7.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs.jar:/usr/lib/hadoop-mapreduce/hadoop-aws-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-s3-1.11.461.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common-2.8.5-amzn-1.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-kms-1.11.461.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry-2.8.5-amzn-1.jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/lib/curator-test-2.7.1.jar:/usr/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.13-tests.jar:/usr/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/lib/hadoop-yarn/lib/audience-annotations-0.5.0.jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.13.jar:/usr/lib/hadoop-yarn/lib/commons-math-2.2.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/lib/hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/lib/hadoop-yarn/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.11.393.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/jcl-over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.29.0.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar:/usr/lib/spark/yarn/lib/datanucleus-core.jar:/usr/lib/spark/yarn/lib/datanucleus-rdbms.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.1.0.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticsearch-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-storagegateway-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-clouddirectory-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacemeteringservice-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-signer-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-snowball-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticbeanstalk-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitosync-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloud9-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3control-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastoredata-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscalingplans-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediapackage-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-secretsmanager-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-transcribe-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-applicationautoscaling-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicediscovery-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediaconvert-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-licensemanager-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplaceentitlement-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-translate-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticache-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-transfer-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-migrationhub-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidentity-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-fms-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-machinelearning-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directory-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elastictranscoder-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpleworkflow-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacecommerceanalytics-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastore-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-xray-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dynamodb-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-connect-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-macie-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lexmodelbuilding-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpointemail-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesisvideo-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ram-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicecatalog-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-serverlessapplicationrepository-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidp-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesisanalyticsv2-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-support-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mq-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-comprehendmedical-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecr-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-waf-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediatailor-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-eks-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-robomaker-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workmail-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costandusagereport-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-stepfunctions-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-guardduty-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-discovery-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpoint-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mobile-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workdocs-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glue-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-code-generator-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-medialive-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot1clickdevices-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-chime-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-organizations-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemakerruntime-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dlm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudformation-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rekognition-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot1clickprojects-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-health-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appmesh-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworkscm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sns-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-greengrass-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servermigration-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-fsx-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-logs-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api-gateway-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-alexaforbusiness-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pi-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-neptune-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-batch-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudtrail-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-amplify-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glacier-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-datasync-1.11.461.jar:/usr/share/aws/aws-java-sdk/jmespath-java-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-quicksight-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-budgets-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemaker-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rdsdata-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dax-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-globalaccelerator-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancing-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroups-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kafka-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecs-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iotanalytics-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpointsmsvoice-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iotjobsdataplane-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworks-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsmv2-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesis-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-datapipeline-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pricing-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lightsail-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediaconnect-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-comprehend-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-devicefarm-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpledb-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudfront-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acmpca-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53resolver-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancingv2-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codepipeline-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-events-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codestar-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatch-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costexplorer-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appsync-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iam-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-securityhub-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kms-1.11.461.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11.461.jar: java.io.tmpdir: /mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tmp user.dir: /mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001 user.name: yarn ************************************************************/ 2019-04-16 11:51:44,193 [INFO] [main] |shim.HadoopShimsLoader|: Trying to locate HadoopShimProvider for hadoopVersion=2.8.5-amzn-1, majorVersion=2, minorVersion=8 2019-04-16 11:51:44,195 [INFO] [main] |shim.HadoopShimsLoader|: Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.8.5-amzn-1, majorVersion=2, minorVersion=8 2019-04-16 11:51:44,217 [INFO] [main] |app.DAGAppMaster|: AM Level configured TaskSchedulers: [0:TezYarn:null],[1:TezUber:null] 2019-04-16 11:51:44,217 [INFO] [main] |app.DAGAppMaster|: AM Level configured ContainerLaunchers: [0:TezYarn:null],[1:TezUber:null] 2019-04-16 11:51:44,217 [INFO] [main] |app.DAGAppMaster|: AM Level configured TaskCommunicators: [0:TezYarn:null],[1:TezUber:null] 2019-04-16 11:51:44,217 [INFO] [main] |app.DAGAppMaster|: Comparing client version with AM version, clientVersion=0.9.1, AMVersion=0.9.1 2019-04-16 11:51:44,788 [INFO] [main] |app.TaskCommunicatorManager|: Creating Default Task Communicator 2019-04-16 11:51:44,794 [INFO] [main] |Configuration.deprecation|: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive 2019-04-16 11:51:44,797 [INFO] [main] |Configuration.deprecation|: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 2019-04-16 11:51:44,798 [INFO] [main] |app.TaskCommunicatorManager|: Creating Default Local Task Communicator 2019-04-16 11:51:44,804 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap 2019-04-16 11:51:44,805 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker 2019-04-16 11:51:44,806 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler 2019-04-16 11:51:44,806 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher 2019-04-16 11:51:44,807 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher 2019-04-16 11:51:44,807 [INFO] [main] |app.DAGAppMaster|: Using concurrent dispatcher: false 2019-04-16 11:51:44,807 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher 2019-04-16 11:51:44,808 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher 2019-04-16 11:51:44,808 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler 2019-04-16 11:51:44,808 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler 2019-04-16 11:51:44,829 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager 2019-04-16 11:51:44,831 [INFO] [main] |launcher.TezContainerLauncherImpl|: Creating DefaultContainerLauncher 2019-04-16 11:51:44,837 [INFO] [main] |launcher.TezContainerLauncherImpl|: Upper limit on the thread pool size is 500 2019-04-16 11:51:44,837 [INFO] [main] |launcher.TezContainerLauncherImpl|: Creating LocalContainerLauncher 2019-04-16 11:51:44,845 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager 2019-04-16 11:51:44,898 [INFO] [main] |node.AMNodeTracker|: blacklistDisablePercent is 33, blacklistingEnabled: true, maxTaskFailuresPerNode: 10, nodeUpdatesRescheduleEnabled: false 2019-04-16 11:51:44,899 [INFO] [main] |web.WebUIService|: Tez UI History URL: http://ip-172-31-90-92.ec2.internal:8080/tez-ui/#/tez-app/application_1555406634668_0016 2019-04-16 11:51:44,900 [INFO] [main] |history.HistoryEventHandler|: Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService 2019-04-16 11:51:45,219 [INFO] [main] |impl.TimelineClientImpl|: Timeline service address: http://ip-172-31-90-92.ec2.internal:8188/ws/v1/timeline/ 2019-04-16 11:51:45,219 [INFO] [main] |ats.ATSHistoryLoggingService|: Initializing ATSHistoryLoggingService with maxEventsPerBatch=5, maxPollingTime(ms)=10, waitTimeForShutdown(ms)=-1, TimelineACLManagerClass=org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager 2019-04-16 11:51:45,326 [INFO] [main] |impl.TimelineClientImpl|: Timeline service address: http://ip-172-31-90-92.ec2.internal:8188/ws/v1/timeline/ 2019-04-16 11:51:45,327 [INFO] [main] |recovery.RecoveryService|: RecoveryService initialized with recoveryPath=hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/.tez/application_1555406634668_0016/recovery/1, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100 2019-04-16 11:51:45,328 [INFO] [main] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:N/A][Event:APP_LAUNCHED]: applicationId=application_1555406634668_0016, appSubmitTime=1555415502182, launchTime=1555415504143 2019-04-16 11:51:45,328 [INFO] [main] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1555406634668_0016_000001, appSubmitTime=1555415502182, launchTime=1555415504143 2019-04-16 11:51:45,347 [INFO] [ServiceThread:org.apache.tez.dag.app.launcher.ContainerLauncherManager] |impl.ContainerManagementProtocolProxy|: yarn.client.max-cached-nodemanagers-proxies : 0 2019-04-16 11:51:45,351 [INFO] [ServiceThread:org.apache.tez.dag.app.TaskCommunicatorManager] |ipc.CallQueueManager|: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 2019-04-16 11:51:45,352 [INFO] [ServiceThread:DAGClientRPCServer] |ipc.CallQueueManager|: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 200 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 2019-04-16 11:51:45,376 [INFO] [Socket Reader #1 for port 37007] |ipc.Server|: Starting Socket Reader #1 for port 37007 2019-04-16 11:51:45,377 [INFO] [Socket Reader #1 for port 46455] |ipc.Server|: Starting Socket Reader #1 for port 46455 2019-04-16 11:51:45,400 [INFO] [IPC Server Responder] |ipc.Server|: IPC Server Responder: starting 2019-04-16 11:51:45,403 [INFO] [IPC Server listener on 37007] |ipc.Server|: IPC Server listener on 37007: starting 2019-04-16 11:51:45,403 [INFO] [IPC Server Responder] |ipc.Server|: IPC Server Responder: starting 2019-04-16 11:51:45,404 [INFO] [IPC Server listener on 46455] |ipc.Server|: IPC Server listener on 46455: starting 2019-04-16 11:51:45,423 [INFO] [ServiceThread:DAGClientRPCServer] |client.DAGClientServer|: Instantiated DAGClientRPCServer at ip-172-31-92-1.ec2.internal/172.31.92.1:37007 2019-04-16 11:51:45,460 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2019-04-16 11:51:45,466 [INFO] [ServiceThread:org.apache.tez.dag.app.TaskCommunicatorManager] |app.TezTaskCommunicatorImpl|: Instantiated TezTaskCommunicator RPC at ip-172-31-92-1.ec2.internal/172.31.92.1:46455 2019-04-16 11:51:45,468 [INFO] [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler] |acls.ATSHistoryACLPolicyManager|: Created Timeline Domain for History ACLs, domainId=Tez_ATS_application_1555406634668_0016 2019-04-16 11:51:45,471 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |server.AuthenticationFilter|: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2019-04-16 11:51:45,476 [WARN] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpRequestLog|: Jetty request log can only be enabled using Log4j 2019-04-16 11:51:45,481 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2019-04-16 11:51:45,484 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context 2019-04-16 11:51:45,484 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static 2019-04-16 11:51:45,487 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: adding path spec: /* 2019-04-16 11:51:45,862 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |webapp.WebApps|: Registered webapp guice modules 2019-04-16 11:51:45,863 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Jetty bound to port 34663 2019-04-16 11:51:45,863 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: jetty-6.1.26 2019-04-16 11:51:45,886 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: Extract jar:file:/usr/lib/hadoop/hadoop-common-2.8.5-amzn-1-tests.jar!/webapps/ to /mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/tmp/Jetty_0_0_0_0_34663_webapps____lh4fwj/webapp 2019-04-16 11:51:46,099 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:34663 2019-04-16 11:51:46,099 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |webapp.WebApps|: Web app started at 34663 2019-04-16 11:51:46,099 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |web.WebUIService|: Instantiated WebUIService at http://ip-172-31-92-1.ec2.internal:34663/ui/ 2019-04-16 11:51:46,101 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.TaskSchedulerManager|: Creating TaskScheduler: YarnTaskSchedulerService 2019-04-16 11:51:46,127 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |Configuration.deprecation|: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 2019-04-16 11:51:46,128 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.TaskSchedulerManager|: Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=111101111 2019-04-16 11:51:46,139 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.YarnTaskSchedulerService|: YarnTaskScheduler initialized with configuration: maxRMHeartbeatInterval: 1000, containerReuseEnabled: true, reuseRackLocal: true, reuseNonLocal: false, localitySchedulingDelay: 250, preemptionPercentage: 10, preemptionMaxWaitTime: 60000, numHeartbeatsBetweenPreemptions: 3, idleContainerMinTimeout: 5000, idleContainerMaxTimeout: 10000, sessionMinHeldContainers: 0 2019-04-16 11:51:46,150 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |client.RMProxy|: Connecting to ResourceManager at ip-172-31-90-92.ec2.internal/172.31.90.92:8030 2019-04-16 11:51:46,231 [INFO] [main] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1555406634668_0016_000001, startTime=1555415506230 2019-04-16 11:51:46,231 [INFO] [main] |app.DAGAppMaster|: In Session mode. Waiting for DAG over RPC 2019-04-16 11:51:46,272 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: App total resource memory: 35840 cpu: 11 taskAllocations: 0 2019-04-16 11:51:46,272 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: Free: pendingRequests: 0 delayedContainers: 0 heartbeats: 1 lastPreemptionHeartbeat: 0 2019-04-16 11:51:46,274 [INFO] [Dispatcher thread {Central}] |node.PerSourceNodeTracker|: Num cluster nodes = 3 2019-04-16 11:51:46,698 [INFO] [IPC Server handler 0 on 37007] |app.DAGAppMaster|: Starting DAG submitted via RPC: insert overwrite table idTable values (1)(Stage-1) 2019-04-16 11:51:46,699 [INFO] [IPC Server handler 0 on 37007] |app.DAGAppMaster|: Merging AM credentials into DAG credentials 2019-04-16 11:51:46,735 [INFO] [IPC Server handler 0 on 37007] |app.DAGAppMaster|: Generating DAG graphviz file, dagId=dag_1555406634668_0016_1, filePath=/var/log/hadoop-yarn/containers/application_1555406634668_0016/container_1555406634668_0016_01_000001/dag_1555406634668_0016_1.dot 2019-04-16 11:51:46,737 [INFO] [IPC Server handler 0 on 37007] |common.TezUtilsInternal|: Redirecting log file based on addend: dag_1555406634668_0016_1 End of LogType:syslog LogType:syslog_dag_1555406634668_0016_1 Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:30574 Log Contents: 2019-04-16 11:51:46,738 [INFO] [IPC Server handler 0 on 37007] |app.DAGAppMaster|: Running DAG: insert overwrite table idTable values (1)(Stage-1), callerContext={ context=HIVE, callerType=HIVE_QUERY_ID, callerId=hive_20190416115136_f07651b2-c968-4408-9a66-eb636ae8a49a } 2019-04-16 11:51:46,837 [INFO] [IPC Server handler 0 on 37007] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_1][Event:DAG_SUBMITTED]: dagID=dag_1555406634668_0016_1, submitTime=1555415506698, queueName=default 2019-04-16 11:51:46,871 [INFO] [HistoryEventHandlingThread] |acls.ATSHistoryACLPolicyManager|: Created Timeline Domain for DAG-specific History ACLs, domainId=Tez_ATS_application_1555406634668_0016_1 2019-04-16 11:51:46,874 [INFO] [IPC Server handler 0 on 37007] |impl.VertexImpl|: Default container context for vertex_1555406634668_0016_1_00 [Map 1]=LocalResources: [[ name=aws-glue-datacatalog-hive2-client.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/aws-glue-datacatalog-hive2-client.jar" } size: 113816 timestamp: 1555415499088 type: FILE visibility: PRIVATE],[ name=emr-hive-goodies.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/emr-hive-goodies.jar" } size: 59642 timestamp: 1555415499512 type: FILE visibility: PRIVATE],[ name=hive-exec-2.3.4-amzn-0-b4b465141bab0a22bba33ac51550e7bd41a2a42cf3dd3f440846c25eb6e4c431.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/user/hadoop/.hiveJars/hive-exec-2.3.4-amzn-0-b4b465141bab0a22bba33ac51550e7bd41a2a42cf3dd3f440846c25eb6e4c431.jar" } size: 34232023 timestamp: 1555415500711 type: FILE visibility: PRIVATE],[ name=tez-conf.pb, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/.tez/application_1555406634668_0016/tez-conf.pb" } size: 94290 timestamp: 1555415501695 type: FILE visibility: APPLICATION],[ name=emr-ddb-hive.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/emr-ddb-hive.jar" } size: 76349 timestamp: 1555415499100 type: FILE visibility: PRIVATE],[ name=tezlib, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/apps/tez/tez.tar.gz" } size: 16279685 timestamp: 1555406393179 type: ARCHIVE visibility: PUBLIC],[ name=hive-hcatalog-server-extensions-2.3.4-amzn-0.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/hive-hcatalog-server-extensions-2.3.4-amzn-0.jar" } size: 73303 timestamp: 1555415499060 type: FILE visibility: PRIVATE],[ name=hive-openx-serde.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/hive-openx-serde.jar" } size: 85727 timestamp: 1555415499948 type: FILE visibility: PRIVATE],[ name=hive-hcatalog-pig-adapter-2.3.4-amzn-0.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/hive-hcatalog-pig-adapter-2.3.4-amzn-0.jar" } size: 53583 timestamp: 1555415498647 type: FILE visibility: PRIVATE],[ name=emr-kinesis-hive.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/emr-kinesis-hive.jar" } size: 8770 timestamp: 1555415499924 type: FILE visibility: PRIVATE],[ name=emr-s3-select-hive-connector.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/emr-s3-select-hive-connector.jar" } size: 20561 timestamp: 1555415499935 type: FILE visibility: PRIVATE],[ name=hive-hcatalog-core-2.3.4-amzn-0.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/hive-hcatalog-core-2.3.4-amzn-0.jar" } size: 264403 timestamp: 1555415498569 type: FILE visibility: PRIVATE],[ name=hive-hcatalog-streaming-2.3.4-amzn-0.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/hive-hcatalog-streaming-2.3.4-amzn-0.jar" } size: 128048 timestamp: 1555415499076 type: FILE visibility: PRIVATE],], environment: [[ SHELL=/bin/bash ],[ LD_LIBRARY_PATH=$PWD:/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:$PWD:$PWD:/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:$PWD:/usr/lib/hadoop/lib/native/ ],[ HADOOP_ROOT_LOGGER=INFO,CLA ],[ CLASSPATH=$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar:/usr/lib/spark/yarn/lib/datanucleus-core.jar:/usr/lib/spark/yarn/lib/datanucleus-rdbms.jar:/usr/share/aws/emr/cloudwatch-sink/lib/*:/usr/share/aws/aws-java-sdk/*: ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir= -Dtez.root.logger=INFO,CLA -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx2458m -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir= -Dtez.root.logger=INFO,CLA , vertex: vertex_1555406634668_0016_1_00 [Map 1], Default Resources= 2019-04-16 11:51:46,874 [INFO] [IPC Server handler 0 on 37007] |impl.VertexImpl|: Setting 1 additional inputs for vertexvertex_1555406634668_0016_1_00 [Map 1] 2019-04-16 11:51:46,874 [INFO] [IPC Server handler 0 on 37007] |impl.VertexImpl|: Setting 1 additional outputs for vertex vertex_1555406634668_0016_1_00 [Map 1] 2019-04-16 11:51:46,875 [INFO] [IPC Server handler 0 on 37007] |impl.VertexImpl|: Using ExecutionContext from Vertex for Vertex Map 1 2019-04-16 11:51:46,876 [INFO] [IPC Server handler 0 on 37007] |impl.VertexImpl|: Running vertex: vertex_1555406634668_0016_1_00 [Map 1] : TaskScheduler=0:TezYarn, ContainerLauncher=0:TezYarn, TaskCommunicator=0:TezYarn 2019-04-16 11:51:46,876 [INFO] [IPC Server handler 0 on 37007] |impl.DAGImpl|: Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrder 2019-04-16 11:51:46,877 [INFO] [IPC Server handler 0 on 37007] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_1][Event:DAG_INITIALIZED]: dagID=dag_1555406634668_0016_1, initTime=1555415506877 2019-04-16 11:51:46,877 [INFO] [IPC Server handler 0 on 37007] |impl.DAGImpl|: dag_1555406634668_0016_1 transitioned from NEW to INITED due to event DAG_INIT 2019-04-16 11:51:46,878 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Added additional resources : [[]] to classpath 2019-04-16 11:51:46,879 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_1][Event:DAG_STARTED]: dagID=dag_1555406634668_0016_1, startTime=1555415506878 2019-04-16 11:51:46,879 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: dag_1555406634668_0016_1 transitioned from INITED to RUNNING due to event DAG_START 2019-04-16 11:51:46,879 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Root Inputs exist for Vertex: Map 1 : {values__tmp__table__1={InputName=values__tmp__table__1}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInputLegacy, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator, hasPayload=false}} 2019-04-16 11:51:46,879 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Starting root input initializer for input: values__tmp__table__1, with class: [org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator] 2019-04-16 11:51:46,879 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Setting vertexManager to RootInputVertexManager for vertex_1555406634668_0016_1_00 [Map 1] 2019-04-16 11:51:46,896 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1555406634668_0016_1_00 [Map 1] 2019-04-16 11:51:46,896 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Vertex will initialize from input initializer. vertex_1555406634668_0016_1_00 [Map 1] 2019-04-16 11:51:46,897 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Starting 1 inputInitializers for vertex vertex_1555406634668_0016_1_00 [Map 1] 2019-04-16 11:51:46,906 [WARN] [HistoryEventHandlingThread] |ats.ATSHistoryLoggingService|: Could not post history event to ATS, atsPutError=6, entityId=dag_1555406634668_0016_1 2019-04-16 11:51:46,906 [WARN] [HistoryEventHandlingThread] |ats.ATSHistoryLoggingService|: Could not post history event to ATS, atsPutError=6, entityId=dag_1555406634668_0016_1 2019-04-16 11:51:46,912 [INFO] [Dispatcher thread {Central}] |Configuration.deprecation|: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed 2019-04-16 11:51:46,957 [INFO] [Dispatcher thread {Central}] |conf.HiveConf|: Found configuration file null 2019-04-16 11:51:46,960 [INFO] [Dispatcher thread {Central}] |tez.HiveSplitGenerator|: SplitGenerator using llap affinitized locations: false 2019-04-16 11:51:46,961 [INFO] [Dispatcher thread {Central}] |tez.HiveSplitGenerator|: SplitLocationProvider: org.apache.hadoop.hive.ql.exec.tez.Utils$1@6cc2abad 2019-04-16 11:51:47,039 [INFO] [Dispatcher thread {Central}] |exec.Utilities|: PLAN PATH = hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/hive_2019-04-16_11-51-36_651_399717790387024222-2/hadoop/_tez_scratch_dir/c2b90fcf-48b3-498b-8ee8-ee021d9875b7/map.xml 2019-04-16 11:51:47,045 [INFO] [Dispatcher thread {Central}] |exec.SerializationUtilities|: Deserializing MapWork using kryo 2019-04-16 11:51:47,116 [INFO] [Dispatcher thread {Central}] |exec.Utilities|: Deserialized plan (via RPC) - name: Map 1 size: 2.19KB 2019-04-16 11:51:47,120 [INFO] [InputInitializer {Map 1} #0] |dag.RootInputInitializerManager|: Starting InputInitializer for Input: values__tmp__table__1 on vertex vertex_1555406634668_0016_1_00 [Map 1] 2019-04-16 11:51:47,121 [INFO] [InputInitializer {Map 1} #0] |tez.HiveSplitGenerator|: GenerateConsistentSplitsInHive=true 2019-04-16 11:51:47,128 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1555406634668_0016_1_00 [Map 1] transitioned from NEW to INITIALIZING due to event V_INIT 2019-04-16 11:51:47,132 [INFO] [InputInitializer {Map 1} #0] |tez.HiveSplitGenerator|: The preferred split size is 52428800 2019-04-16 11:51:47,132 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: PLAN PATH = hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/hive_2019-04-16_11-51-36_651_399717790387024222-2/hadoop/_tez_scratch_dir/c2b90fcf-48b3-498b-8ee8-ee021d9875b7/map.xml 2019-04-16 11:51:47,133 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: Processing alias values__tmp__table__1 2019-04-16 11:51:47,133 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: Adding 1 inputs; the first input is hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/_tmp_space.db/Values__Tmp__Table__1 2019-04-16 11:51:47,135 [INFO] [InputInitializer {Map 1} #0] |io.HiveInputFormat|: hive.io.file.readcolumn.ids = 0 2019-04-16 11:51:47,135 [INFO] [InputInitializer {Map 1} #0] |io.HiveInputFormat|: hive.io.file.readcolumn.names = tmp_values_col1 2019-04-16 11:51:47,135 [INFO] [InputInitializer {Map 1} #0] |io.HiveInputFormat|: Generating splits for dirs: hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/_tmp_space.db/Values__Tmp__Table__1 2019-04-16 11:51:47,145 [ERROR] [InputInitializer {Map 1} #0] |lzo.GPLNativeCodeLoader|: Could not load native gpl library java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) at java.lang.Runtime.loadLibrary0(Runtime.java:870) at java.lang.System.loadLibrary(System.java:1122) at com.hadoop.compression.lzo.GPLNativeCodeLoader.(GPLNativeCodeLoader.java:32) at com.hadoop.compression.lzo.LzoCodec.(LzoCodec.java:71) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2306) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2271) at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:132) at org.apache.hadoop.io.compress.CompressionCodecFactory.(CompressionCodecFactory.java:180) at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hive.common.util.ReflectionUtil.setJobConf(ReflectionUtil.java:112) at org.apache.hive.common.util.ReflectionUtil.setConf(ReflectionUtil.java:103) at org.apache.hive.common.util.ReflectionUtil.newInstance(ReflectionUtil.java:87) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputFormatFromCache(HiveInputFormat.java:322) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:562) at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-04-16 11:51:47,147 [ERROR] [InputInitializer {Map 1} #0] |lzo.LzoCodec|: Cannot load native-lzo without native-hadoop 2019-04-16 11:51:47,176 [INFO] [InputInitializer {Map 1} #0] |mapred.FileInputFormat|: Total input files to process : 1 2019-04-16 11:51:47,180 [INFO] [InputInitializer {Map 1} #0] |io.HiveInputFormat|: number of splits 1 2019-04-16 11:51:47,180 [INFO] [InputInitializer {Map 1} #0] |tez.HiveSplitGenerator|: Number of input splits: 1. 11 available slots, 1.7 waves. Input format is: org.apache.hadoop.hive.ql.io.HiveInputFormat 2019-04-16 11:51:47,180 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: PLAN PATH = hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/hive_2019-04-16_11-51-36_651_399717790387024222-2/hadoop/_tez_scratch_dir/c2b90fcf-48b3-498b-8ee8-ee021d9875b7/map.xml 2019-04-16 11:51:47,180 [INFO] [InputInitializer {Map 1} #0] |exec.SerializationUtilities|: Deserializing MapWork using kryo 2019-04-16 11:51:47,183 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: Deserialized plan (via RPC) - name: Map 1 size: 2.19KB 2019-04-16 11:51:47,184 [INFO] [InputInitializer {Map 1} #0] |tez.SplitGrouper|: # Src groups for split generation: 2 2019-04-16 11:51:47,186 [INFO] [InputInitializer {Map 1} #0] |tez.SplitGrouper|: Estimated number of tasks: 18 for bucket 1 2019-04-16 11:51:47,187 [INFO] [InputInitializer {Map 1} #0] |grouper.TezSplitGrouper|: Grouping splits in Tez 2019-04-16 11:51:47,187 [INFO] [InputInitializer {Map 1} #0] |grouper.TezSplitGrouper|: Desired splits: 1 too large. Desired splitLength: 0 Min splitLength: 52428800 New desired splits: 1 Final desired splits: 1 All splits have localhost: false Total length: 2 Original splits: 1 2019-04-16 11:51:47,188 [INFO] [InputInitializer {Map 1} #0] |grouper.TezSplitGrouper|: Using original number of splits: 1 desired splits: 1 2019-04-16 11:51:47,189 [INFO] [InputInitializer {Map 1} #0] |tez.SplitGrouper|: Original split count is 1 grouped split count is 1, for bucket: 1 2019-04-16 11:51:47,190 [INFO] [InputInitializer {Map 1} #0] |tez.HiveSplitGenerator|: Number of split groups: 1 2019-04-16 11:51:47,198 [INFO] [InputInitializer {Map 1} #0] |dag.RootInputInitializerManager|: Succeeded InputInitializer for Input: values__tmp__table__1 on vertex vertex_1555406634668_0016_1_00 [Map 1] 2019-04-16 11:51:47,199 [INFO] [App Shared Pool - #0] |impl.VertexImpl|: Cannot init vertex: vertex_1555406634668_0016_1_00 [Map 1] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true 2019-04-16 11:51:47,199 [INFO] [App Shared Pool - #0] |impl.VertexImpl|: Got updated RootInputsSpecs: {values__tmp__table__1=forAllWorkUnits=true, update=[1]} 2019-04-16 11:51:47,206 [INFO] [App Shared Pool - #0] |impl.VertexImpl|: Vertex vertex_1555406634668_0016_1_00 [Map 1] parallelism set to 1 2019-04-16 11:51:47,206 [INFO] [App Shared Pool - #0] |impl.VertexImpl|: Cannot init vertex: vertex_1555406634668_0016_1_00 [Map 1] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true 2019-04-16 11:51:47,216 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Setting up committers for vertex vertex_1555406634668_0016_1_00 [Map 1], numAdditionalOutputs=1 2019-04-16 11:51:47,216 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_1][Event:VERTEX_INITIALIZED]: vertexName=Map 1, vertexId=vertex_1555406634668_0016_1_00, initRequestedTime=1555415506879, initedTime=1555415507216, numTasks=1, processorName=org.apache.hadoop.hive.ql.exec.tez.MapTezProcessor, additionalInputsCount=1, initGeneratedEventsCount=1, servicePluginInfo=ServicePluginInfo {containerLauncherName=TezYarn, taskSchedulerName=TezYarn, taskCommunicatorName=TezYarn, containerLauncherClassName=org.apache.tez.dag.app.launcher.TezContainerLauncherImpl, taskSchedulerClassName=org.apache.tez.dag.app.rm.YarnTaskSchedulerService, taskCommunicatorClassName=org.apache.tez.dag.app.TezTaskCommunicatorImpl } 2019-04-16 11:51:47,216 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Triggering start event for vertex: vertex_1555406634668_0016_1_00 [Map 1] with distanceFromRoot: 0 2019-04-16 11:51:47,217 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1555406634668_0016_1_00 [Map 1] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION 2019-04-16 11:51:47,217 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_1][Event:VERTEX_STARTED]: vertexId=vertex_1555406634668_0016_1_00, startRequestedTime=1555415507134, startedTime=1555415507217 2019-04-16 11:51:47,218 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1555406634668_0016_1_00, reconfigureDoneTime=1555415507218, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@49529cd4, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true 2019-04-16 11:51:47,218 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1555406634668_0016_1_00 [Map 1] transitioned from INITED to RUNNING due to event V_START 2019-04-16 11:51:47,219 [INFO] [App Shared Pool - #0] |impl.RootInputVertexManager|: All source tasks completed. Ramping up 1 remaining tasks for vertex: Map 1 2019-04-16 11:51:47,224 [INFO] [App Shared Pool - #0] |impl.RootInputVertexManager|: Scheduling 1 tasks for vertex: Map 1 with totalTasks: 1. 0 source tasks completed out of 0. MinSourceTaskCompletedFraction: 1.0 min: 0.0 max: 0.0 2019-04-16 11:51:47,225 [INFO] [App Shared Pool - #0] |impl.VertexImpl|: Routing pending task events for vertex: vertex_1555406634668_0016_1_00 [Map 1] 2019-04-16 11:51:47,228 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_1][Event:TASK_STARTED]: vertexName=Map 1, taskId=task_1555406634668_0016_1_00_000000, scheduledTime=1555415507227, launchTime=1555415507227 2019-04-16 11:51:47,252 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1555406634668_0016_1_00_000000_0 with request: Capability[]Priority[2] host: ip-172-31-92-1.ec2.internal rack: null 2019-04-16 11:51:48,281 [INFO] [AMRM Heartbeater thread] |impl.AMRMClientImpl|: Received new token for : ip-172-31-91-60.ec2.internal:8041 2019-04-16 11:51:48,539 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_1555406634668_0016_01_000002, task=attempt_1555406634668_0016_1_00_000000_0, containerHost=ip-172-31-91-60.ec2.internal:8041, containerPriority= 2, containerResources=, localityMatchType=RackLocal, matchedLocation=/default-rack, honorLocalityFlags=false, reusedContainer=false, delayedContainers=0 2019-04-16 11:51:48,552 [INFO] [TaskSchedulerAppCallbackExecutor #0] |node.PerSourceNodeTracker|: Adding new node ip-172-31-91-60.ec2.internal:8041 to nodeTracker 0 2019-04-16 11:51:48,568 [INFO] [ContainerLauncher #0] |launcher.TezContainerLauncherImpl|: Launching container_1555406634668_0016_01_000002 2019-04-16 11:51:48,626 [INFO] [ContainerLauncher #0] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1555406634668_0016_01_000002, launchTime=1555415508626 2019-04-16 11:51:49,957 [INFO] [Socket Reader #1 for port 46455] |ipc.Server|: Auth successful for application_1555406634668_0016 (auth:SIMPLE) 2019-04-16 11:51:49,978 [INFO] [Dispatcher thread {Central}] |impl.TaskAttemptImpl|: TaskAttempt: [attempt_1555406634668_0016_1_00_000000_0] submitted. Is using containerId: [container_1555406634668_0016_01_000002] on NM: [ip-172-31-91-60.ec2.internal:8041] 2019-04-16 11:51:49,978 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_1][Event:TASK_ATTEMPT_STARTED]: vertexName=Map 1, taskAttemptId=attempt_1555406634668_0016_1_00_000000_0, startTime=1555415509977, containerId=container_1555406634668_0016_01_000002, nodeId=ip-172-31-91-60.ec2.internal:8041 2019-04-16 11:51:49,979 [INFO] [Dispatcher thread {Central}] |counters.Limits|: Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200 2019-04-16 11:51:50,071 [INFO] [IPC Server handler 1 on 46455] |impl.TaskImpl|: TaskAttempt:attempt_1555406634668_0016_1_00_000000_0 sent events: (0-1). 2019-04-16 11:51:52,026 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=Map 1, taskAttemptId=attempt_1555406634668_0016_1_00_000000_0, creationTime=1555415507239, allocationTime=1555415508565, startTime=1555415509977, finishTime=1555415512025, timeTaken=2048, status=SUCCEEDED, diagnostics= 2019-04-16 11:51:52,027 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Task succeeded with attempt attempt_1555406634668_0016_1_00_000000_0 2019-04-16 11:51:52,031 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_1][Event:TASK_FINISHED]: vertexName=Map 1, taskId=task_1555406634668_0016_1_00_000000, startTime=1555415509977, finishTime=1555415512027, timeTaken=2050, status=SUCCEEDED, successfulAttemptID=attempt_1555406634668_0016_1_00_000000_0, diagnostics=, counters=Counters: 18, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, HDFS_BYTES_READ=2, HDFS_BYTES_WRITTEN=73, HDFS_READ_OPS=4, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=3, org.apache.tez.common.counters.TaskCounter, GC_TIME_MILLIS=72, CPU_MILLISECONDS=2910, PHYSICAL_MEMORY_BYTES=315621376, VIRTUAL_MEMORY_BYTES=4534431744, COMMITTED_HEAP_BYTES=315621376, INPUT_RECORDS_PROCESSED=1, INPUT_SPLIT_LENGTH_BYTES=2, OUTPUT_RECORDS=0, HIVE, CREATED_FILES=1, DESERIALIZE_ERRORS=0, RECORDS_IN_Map_1=1, RECORDS_OUT_1_default.idtable=1 2019-04-16 11:51:52,034 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Task Completion: vertex_1555406634668_0016_1_00 [Map 1], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null 2019-04-16 11:51:52,034 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: All tasks have succeeded, vertex:vertex_1555406634668_0016_1_00 [Map 1] 2019-04-16 11:51:52,042 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_1][Event:VERTEX_FINISHED]: vertexName=Map 1, vertexId=vertex_1555406634668_0016_1_00, initRequestedTime=1555415506879, initedTime=1555415507216, startRequestedTime=1555415507134, startedTime=1555415507217, finishTime=1555415512034, timeTaken=4817, status=SUCCEEDED, diagnostics=, counters=Counters: 18, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, HDFS_BYTES_READ=2, HDFS_BYTES_WRITTEN=73, HDFS_READ_OPS=4, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=3, org.apache.tez.common.counters.TaskCounter, GC_TIME_MILLIS=72, CPU_MILLISECONDS=2910, PHYSICAL_MEMORY_BYTES=315621376, VIRTUAL_MEMORY_BYTES=4534431744, COMMITTED_HEAP_BYTES=315621376, INPUT_RECORDS_PROCESSED=1, INPUT_SPLIT_LENGTH_BYTES=2, OUTPUT_RECORDS=0, HIVE, CREATED_FILES=1, DESERIALIZE_ERRORS=0, RECORDS_IN_Map_1=1, RECORDS_OUT_1_default.idtable=1, vertexStats=firstTaskStartTime=1555415509977, firstTasksToStart=[ task_1555406634668_0016_1_00_000000 ], lastTaskFinishTime=1555415512025, lastTasksToFinish=[ task_1555406634668_0016_1_00_000000 ], minTaskDuration=2048, maxTaskDuration=2048, avgTaskDuration=2048.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1555406634668_0016_1_00_000000 ], longestDurationTasks=[ task_1555406634668_0016_1_00_000000 ], vertexTaskStats={numFailedTaskAttempts=0, numKilledTaskAttempts=0, numCompletedTasks=1, numSucceededTasks=1, numKilledTasks=0, numFailedTasks=0}, servicePluginInfo=ServicePluginInfo {containerLauncherName=TezYarn, taskSchedulerName=TezYarn, taskCommunicatorName=TezYarn, containerLauncherClassName=org.apache.tez.dag.app.launcher.TezContainerLauncherImpl, taskSchedulerClassName=org.apache.tez.dag.app.rm.YarnTaskSchedulerService, taskCommunicatorClassName=org.apache.tez.dag.app.TezTaskCommunicatorImpl } 2019-04-16 11:51:52,043 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1555406634668_0016_1_00 [Map 1] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED 2019-04-16 11:51:52,043 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Vertex vertex_1555406634668_0016_1_00 [Map 1] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=1 2019-04-16 11:51:52,043 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=1, commitInProgress=0, terminationCause=null 2019-04-16 11:51:52,043 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: No exclusive output committers for vertex: vertex_1555406634668_0016_1_00 [Map 1] 2019-04-16 11:51:52,048 [INFO] [Dispatcher thread {Central}] |recovery.RecoveryService|: DAG completed, dagId=dag_1555406634668_0016_1, queueSize=0 2019-04-16 11:51:52,055 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_1][Event:DAG_FINISHED]: dagId=dag_1555406634668_0016_1, startTime=1555415506878, finishTime=1555415512043, timeTaken=5165, status=SUCCEEDED, diagnostics=, counters=Counters: 22, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=1, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=1920, AM_GC_TIME_MILLIS=0, File System Counters, HDFS_BYTES_READ=2, HDFS_BYTES_WRITTEN=73, HDFS_READ_OPS=4, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=3, org.apache.tez.common.counters.TaskCounter, GC_TIME_MILLIS=72, CPU_MILLISECONDS=2910, PHYSICAL_MEMORY_BYTES=315621376, VIRTUAL_MEMORY_BYTES=4534431744, COMMITTED_HEAP_BYTES=315621376, INPUT_RECORDS_PROCESSED=1, INPUT_SPLIT_LENGTH_BYTES=2, OUTPUT_RECORDS=0, HIVE, CREATED_FILES=1, DESERIALIZE_ERRORS=0, RECORDS_IN_Map_1=1, RECORDS_OUT_1_default.idtable=1 2019-04-16 11:51:52,055 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: DAG: dag_1555406634668_0016_1 finished with state: SUCCEEDED 2019-04-16 11:51:52,055 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: dag_1555406634668_0016_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED 2019-04-16 11:51:52,056 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: DAG completed, dagId=dag_1555406634668_0016_1, dagState=SUCCEEDED 2019-04-16 11:51:52,056 [INFO] [Dispatcher thread {Central}] |common.TezUtilsInternal|: Redirecting log file based on addend: dag_1555406634668_0016_1_post End of LogType:syslog_dag_1555406634668_0016_1 LogType:syslog_dag_1555406634668_0016_1_post Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:2984 Log Contents: 2019-04-16 11:51:52,056 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Central Dispatcher queue size after DAG completion, before cleanup: 0 2019-04-16 11:51:52,056 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Waiting for next DAG to be submitted. 2019-04-16 11:51:52,056 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Cleaning up DAG: name=insert overwrite table idTable values (1)(Stage-1), with id=dag_1555406634668_0016_1 2019-04-16 11:51:52,056 [INFO] [Dispatcher thread {Central}] |container.AMContainerMap|: Cleaned up completed containers on dagComplete. Removed=0, Remaining=1 2019-04-16 11:51:52,056 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Completed cleanup for DAG: name=insert overwrite table idTable values (1)(Stage-1), with id=dag_1555406634668_0016_1 2019-04-16 11:52:00,281 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: No taskRequests. Container's idle timeout delay expired or is new. Releasing container, containerId=container_1555406634668_0016_01_000002, containerExpiryTime=1555415520216, idleTimeout=5000, taskRequestsCount=0, heldContainers=1, delayedContainers=0, isNew=false 2019-04-16 11:52:00,283 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_1][Event:CONTAINER_STOPPED]: containerId=container_1555406634668_0016_01_000002, stoppedTime=1555415520283, exitStatus=0 2019-04-16 11:52:00,284 [INFO] [ContainerLauncher #1] |launcher.TezContainerLauncherImpl|: Stopping container_1555406634668_0016_01_000002 2019-04-16 11:52:00,284 [INFO] [IPC Server handler 18 on 46455] |app.TezTaskCommunicatorImpl|: Container with id: container_1555406634668_0016_01_000002 is valid, but no longer registered, and will be killed 2019-04-16 11:52:00,310 [INFO] [Dispatcher thread {Central}] |container.AMContainerImpl|: Container container_1555406634668_0016_01_000002 exited with diagnostics set to Container failed, exitCode=-100. Container released by application 2019-04-16 11:52:36,363 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: Free: pendingRequests: 0 delayedContainers: 0 heartbeats: 51 lastPreemptionHeartbeat: 50 2019-04-16 11:53:09,589 [INFO] [IPC Server handler 1 on 37007] |app.DAGAppMaster|: Starting DAG submitted via RPC: Select T.Candidate.startTime, T.Candidat...T(Stage-1) 2019-04-16 11:53:09,589 [INFO] [IPC Server handler 1 on 37007] |app.DAGAppMaster|: Merging AM credentials into DAG credentials 2019-04-16 11:53:09,590 [INFO] [IPC Server handler 1 on 37007] |app.DAGAppMaster|: Generating DAG graphviz file, dagId=dag_1555406634668_0016_2, filePath=/var/log/hadoop-yarn/containers/application_1555406634668_0016/container_1555406634668_0016_01_000001/dag_1555406634668_0016_2.dot 2019-04-16 11:53:09,591 [INFO] [IPC Server handler 1 on 37007] |common.TezUtilsInternal|: Redirecting log file based on addend: dag_1555406634668_0016_2 End of LogType:syslog_dag_1555406634668_0016_1_post LogType:syslog_dag_1555406634668_0016_2 Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:27487 Log Contents: 2019-04-16 11:53:09,591 [INFO] [IPC Server handler 1 on 37007] |app.DAGAppMaster|: Running DAG: Select T.Candidate.startTime, T.Candidat...T(Stage-1), callerContext={ context=HIVE, callerType=HIVE_QUERY_ID, callerId=hive_20190416115250_eab01a63-574a-4628-a799-bd359db0122c } 2019-04-16 11:53:09,606 [INFO] [IPC Server handler 1 on 37007] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_2][Event:DAG_SUBMITTED]: dagID=dag_1555406634668_0016_2, submitTime=1555415589589, queueName=default 2019-04-16 11:53:09,606 [INFO] [IPC Server handler 1 on 37007] |app.DAGAppMaster|: Localizing additional local resources for AM : {TimestampUDF-1.0-SNAPSHOT.jar=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/TimestampUDF-1.0-SNAPSHOT.jar" } size: 3867 timestamp: 1555415589557 type: FILE visibility: PRIVATE} 2019-04-16 11:53:09,626 [INFO] [HistoryEventHandlingThread] |acls.ATSHistoryACLPolicyManager|: Created Timeline Domain for DAG-specific History ACLs, domainId=Tez_ATS_application_1555406634668_0016_2 2019-04-16 11:53:09,650 [WARN] [HistoryEventHandlingThread] |ats.ATSHistoryLoggingService|: Could not post history event to ATS, atsPutError=6, entityId=dag_1555406634668_0016_2 2019-04-16 11:53:09,650 [WARN] [HistoryEventHandlingThread] |ats.ATSHistoryLoggingService|: Could not post history event to ATS, atsPutError=6, entityId=dag_1555406634668_0016_2 2019-04-16 11:53:09,686 [INFO] [IPC Server handler 1 on 37007] |app.DAGAppMaster|: Done downloading additional AM resources 2019-04-16 11:53:09,694 [INFO] [IPC Server handler 1 on 37007] |impl.VertexImpl|: Default container context for vertex_1555406634668_0016_2_00 [Map 1]=LocalResources: [[ name=aws-glue-datacatalog-hive2-client.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/aws-glue-datacatalog-hive2-client.jar" } size: 113816 timestamp: 1555415499088 type: FILE visibility: PRIVATE],[ name=emr-hive-goodies.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/emr-hive-goodies.jar" } size: 59642 timestamp: 1555415499512 type: FILE visibility: PRIVATE],[ name=hive-exec-2.3.4-amzn-0-b4b465141bab0a22bba33ac51550e7bd41a2a42cf3dd3f440846c25eb6e4c431.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/user/hadoop/.hiveJars/hive-exec-2.3.4-amzn-0-b4b465141bab0a22bba33ac51550e7bd41a2a42cf3dd3f440846c25eb6e4c431.jar" } size: 34232023 timestamp: 1555415500711 type: FILE visibility: PRIVATE],[ name=tez-conf.pb, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/.tez/application_1555406634668_0016/tez-conf.pb" } size: 94290 timestamp: 1555415501695 type: FILE visibility: APPLICATION],[ name=emr-ddb-hive.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/emr-ddb-hive.jar" } size: 76349 timestamp: 1555415499100 type: FILE visibility: PRIVATE],[ name=tezlib, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/apps/tez/tez.tar.gz" } size: 16279685 timestamp: 1555406393179 type: ARCHIVE visibility: PUBLIC],[ name=hive-hcatalog-server-extensions-2.3.4-amzn-0.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/hive-hcatalog-server-extensions-2.3.4-amzn-0.jar" } size: 73303 timestamp: 1555415499060 type: FILE visibility: PRIVATE],[ name=hive-openx-serde.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/hive-openx-serde.jar" } size: 85727 timestamp: 1555415499948 type: FILE visibility: PRIVATE],[ name=hive-hcatalog-pig-adapter-2.3.4-amzn-0.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/hive-hcatalog-pig-adapter-2.3.4-amzn-0.jar" } size: 53583 timestamp: 1555415498647 type: FILE visibility: PRIVATE],[ name=emr-kinesis-hive.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/emr-kinesis-hive.jar" } size: 8770 timestamp: 1555415499924 type: FILE visibility: PRIVATE],[ name=emr-s3-select-hive-connector.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/emr-s3-select-hive-connector.jar" } size: 20561 timestamp: 1555415499935 type: FILE visibility: PRIVATE],[ name=TimestampUDF-1.0-SNAPSHOT.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/TimestampUDF-1.0-SNAPSHOT.jar" } size: 3867 timestamp: 1555415589557 type: FILE visibility: PRIVATE],[ name=hive-hcatalog-core-2.3.4-amzn-0.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/hive-hcatalog-core-2.3.4-amzn-0.jar" } size: 264403 timestamp: 1555415498569 type: FILE visibility: PRIVATE],[ name=hive-hcatalog-streaming-2.3.4-amzn-0.jar, value=resource { scheme: "hdfs" host: "ip-172-31-90-92.ec2.internal" port: 8020 file: "/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/hive-hcatalog-streaming-2.3.4-amzn-0.jar" } size: 128048 timestamp: 1555415499076 type: FILE visibility: PRIVATE],], environment: [[ SHELL=/bin/bash ],[ LD_LIBRARY_PATH=$PWD:/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:$PWD:$PWD:/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:$PWD:/usr/lib/hadoop/lib/native/ ],[ HADOOP_ROOT_LOGGER=INFO,CLA ],[ CLASSPATH=$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar:/usr/lib/spark/yarn/lib/datanucleus-core.jar:/usr/lib/spark/yarn/lib/datanucleus-rdbms.jar:/usr/share/aws/emr/cloudwatch-sink/lib/*:/usr/share/aws/aws-java-sdk/*: ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir= -Dtez.root.logger=INFO,CLA -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx2458m -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir= -Dtez.root.logger=INFO,CLA , vertex: vertex_1555406634668_0016_2_00 [Map 1], Default Resources= 2019-04-16 11:53:09,694 [INFO] [IPC Server handler 1 on 37007] |impl.VertexImpl|: Setting 1 additional inputs for vertexvertex_1555406634668_0016_2_00 [Map 1] 2019-04-16 11:53:09,694 [INFO] [IPC Server handler 1 on 37007] |impl.VertexImpl|: Setting 1 additional outputs for vertex vertex_1555406634668_0016_2_00 [Map 1] 2019-04-16 11:53:09,694 [INFO] [IPC Server handler 1 on 37007] |impl.VertexImpl|: Using ExecutionContext from Vertex for Vertex Map 1 2019-04-16 11:53:09,694 [INFO] [IPC Server handler 1 on 37007] |impl.VertexImpl|: Running vertex: vertex_1555406634668_0016_2_00 [Map 1] : TaskScheduler=0:TezYarn, ContainerLauncher=0:TezYarn, TaskCommunicator=0:TezYarn 2019-04-16 11:53:09,694 [INFO] [IPC Server handler 1 on 37007] |impl.DAGImpl|: Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrder 2019-04-16 11:53:09,694 [INFO] [IPC Server handler 1 on 37007] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_2][Event:DAG_INITIALIZED]: dagID=dag_1555406634668_0016_2, initTime=1555415589694 2019-04-16 11:53:09,696 [INFO] [IPC Server handler 1 on 37007] |impl.DAGImpl|: dag_1555406634668_0016_2 transitioned from NEW to INITED due to event DAG_INIT 2019-04-16 11:53:09,697 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Added additional resources : [[file:/mnt/yarn/usercache/hadoop/appcache/application_1555406634668_0016/container_1555406634668_0016_01_000001/TimestampUDF-1.0-SNAPSHOT.jar]] to classpath 2019-04-16 11:53:09,697 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_2][Event:DAG_STARTED]: dagID=dag_1555406634668_0016_2, startTime=1555415589697 2019-04-16 11:53:09,697 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: dag_1555406634668_0016_2 transitioned from INITED to RUNNING due to event DAG_START 2019-04-16 11:53:09,697 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Root Inputs exist for Vertex: Map 1 : {idtable={InputName=idtable}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInputLegacy, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator, hasPayload=false}} 2019-04-16 11:53:09,697 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Starting root input initializer for input: idtable, with class: [org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator] 2019-04-16 11:53:09,697 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Setting vertexManager to RootInputVertexManager for vertex_1555406634668_0016_2_00 [Map 1] 2019-04-16 11:53:09,705 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1555406634668_0016_2_00 [Map 1] 2019-04-16 11:53:09,705 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Vertex will initialize from input initializer. vertex_1555406634668_0016_2_00 [Map 1] 2019-04-16 11:53:09,705 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Starting 1 inputInitializers for vertex vertex_1555406634668_0016_2_00 [Map 1] 2019-04-16 11:53:09,708 [INFO] [Dispatcher thread {Central}] |tez.HiveSplitGenerator|: SplitGenerator using llap affinitized locations: false 2019-04-16 11:53:09,708 [INFO] [Dispatcher thread {Central}] |tez.HiveSplitGenerator|: SplitLocationProvider: org.apache.hadoop.hive.ql.exec.tez.Utils$1@613b575f 2019-04-16 11:53:09,708 [INFO] [Dispatcher thread {Central}] |exec.Utilities|: PLAN PATH = hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/hive_2019-04-16_11-52-50_757_5369209958768659300-3/hadoop/_tez_scratch_dir/35664f3e-1811-4541-ba57-f45dad3372f5/map.xml 2019-04-16 11:53:09,709 [INFO] [Dispatcher thread {Central}] |exec.SerializationUtilities|: Deserializing MapWork using kryo 2019-04-16 11:53:09,718 [INFO] [Dispatcher thread {Central}] |exec.Utilities|: Deserialized plan (via RPC) - name: Map 1 size: 2.33KB 2019-04-16 11:53:09,720 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1555406634668_0016_2_00 [Map 1] transitioned from NEW to INITIALIZING due to event V_INIT 2019-04-16 11:53:09,721 [INFO] [InputInitializer {Map 1} #0] |dag.RootInputInitializerManager|: Starting InputInitializer for Input: idtable on vertex vertex_1555406634668_0016_2_00 [Map 1] 2019-04-16 11:53:09,724 [INFO] [InputInitializer {Map 1} #0] |tez.HiveSplitGenerator|: GenerateConsistentSplitsInHive=true 2019-04-16 11:53:09,724 [INFO] [InputInitializer {Map 1} #0] |tez.HiveSplitGenerator|: The preferred split size is 52428800 2019-04-16 11:53:09,724 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: PLAN PATH = hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/hive_2019-04-16_11-52-50_757_5369209958768659300-3/hadoop/_tez_scratch_dir/35664f3e-1811-4541-ba57-f45dad3372f5/map.xml 2019-04-16 11:53:09,724 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: Processing alias idtable 2019-04-16 11:53:09,724 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: Adding 1 inputs; the first input is hdfs://ip-172-31-90-92.ec2.internal:8020/user/hive/warehouse/idtable 2019-04-16 11:53:09,725 [INFO] [InputInitializer {Map 1} #0] |io.HiveInputFormat|: hive.io.file.readcolumn.ids = 2019-04-16 11:53:09,725 [INFO] [InputInitializer {Map 1} #0] |io.HiveInputFormat|: hive.io.file.readcolumn.names = 2019-04-16 11:53:09,725 [INFO] [InputInitializer {Map 1} #0] |io.HiveInputFormat|: Generating splits for dirs: hdfs://ip-172-31-90-92.ec2.internal:8020/user/hive/warehouse/idtable 2019-04-16 11:53:09,729 [INFO] [InputInitializer {Map 1} #0] |mapred.FileInputFormat|: Total input files to process : 1 2019-04-16 11:53:09,730 [INFO] [InputInitializer {Map 1} #0] |io.HiveInputFormat|: number of splits 1 2019-04-16 11:53:09,730 [INFO] [InputInitializer {Map 1} #0] |tez.HiveSplitGenerator|: Number of input splits: 1. 11 available slots, 1.7 waves. Input format is: org.apache.hadoop.hive.ql.io.HiveInputFormat 2019-04-16 11:53:09,730 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: PLAN PATH = hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/f266da94-38f8-4b55-9210-35d84558222f/hive_2019-04-16_11-52-50_757_5369209958768659300-3/hadoop/_tez_scratch_dir/35664f3e-1811-4541-ba57-f45dad3372f5/map.xml 2019-04-16 11:53:09,730 [INFO] [InputInitializer {Map 1} #0] |exec.SerializationUtilities|: Deserializing MapWork using kryo 2019-04-16 11:53:09,732 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: Deserialized plan (via RPC) - name: Map 1 size: 2.33KB 2019-04-16 11:53:09,732 [INFO] [InputInitializer {Map 1} #0] |tez.SplitGrouper|: # Src groups for split generation: 2 2019-04-16 11:53:09,732 [INFO] [InputInitializer {Map 1} #0] |tez.SplitGrouper|: Estimated number of tasks: 18 for bucket 1 2019-04-16 11:53:09,733 [INFO] [InputInitializer {Map 1} #0] |grouper.TezSplitGrouper|: Grouping splits in Tez 2019-04-16 11:53:09,733 [INFO] [InputInitializer {Map 1} #0] |grouper.TezSplitGrouper|: Desired splits: 1 too large. Desired splitLength: 0 Min splitLength: 52428800 New desired splits: 1 Final desired splits: 1 All splits have localhost: false Total length: 2 Original splits: 1 2019-04-16 11:53:09,733 [INFO] [InputInitializer {Map 1} #0] |grouper.TezSplitGrouper|: Using original number of splits: 1 desired splits: 1 2019-04-16 11:53:09,733 [INFO] [InputInitializer {Map 1} #0] |tez.SplitGrouper|: Original split count is 1 grouped split count is 1, for bucket: 1 2019-04-16 11:53:09,733 [INFO] [InputInitializer {Map 1} #0] |tez.HiveSplitGenerator|: Number of split groups: 1 2019-04-16 11:53:09,734 [INFO] [InputInitializer {Map 1} #0] |dag.RootInputInitializerManager|: Succeeded InputInitializer for Input: idtable on vertex vertex_1555406634668_0016_2_00 [Map 1] 2019-04-16 11:53:09,735 [INFO] [App Shared Pool - #1] |impl.VertexImpl|: Cannot init vertex: vertex_1555406634668_0016_2_00 [Map 1] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true 2019-04-16 11:53:09,735 [INFO] [App Shared Pool - #1] |impl.VertexImpl|: Got updated RootInputsSpecs: {idtable=forAllWorkUnits=true, update=[1]} 2019-04-16 11:53:09,735 [INFO] [App Shared Pool - #1] |impl.VertexImpl|: Vertex vertex_1555406634668_0016_2_00 [Map 1] parallelism set to 1 2019-04-16 11:53:09,735 [INFO] [App Shared Pool - #1] |impl.VertexImpl|: Cannot init vertex: vertex_1555406634668_0016_2_00 [Map 1] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true 2019-04-16 11:53:09,736 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Setting up committers for vertex vertex_1555406634668_0016_2_00 [Map 1], numAdditionalOutputs=1 2019-04-16 11:53:09,736 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_2][Event:VERTEX_INITIALIZED]: vertexName=Map 1, vertexId=vertex_1555406634668_0016_2_00, initRequestedTime=1555415589697, initedTime=1555415589736, numTasks=1, processorName=org.apache.hadoop.hive.ql.exec.tez.MapTezProcessor, additionalInputsCount=1, initGeneratedEventsCount=1, servicePluginInfo=ServicePluginInfo {containerLauncherName=TezYarn, taskSchedulerName=TezYarn, taskCommunicatorName=TezYarn, containerLauncherClassName=org.apache.tez.dag.app.launcher.TezContainerLauncherImpl, taskSchedulerClassName=org.apache.tez.dag.app.rm.YarnTaskSchedulerService, taskCommunicatorClassName=org.apache.tez.dag.app.TezTaskCommunicatorImpl } 2019-04-16 11:53:09,736 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Triggering start event for vertex: vertex_1555406634668_0016_2_00 [Map 1] with distanceFromRoot: 0 2019-04-16 11:53:09,736 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1555406634668_0016_2_00 [Map 1] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION 2019-04-16 11:53:09,736 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_2][Event:VERTEX_STARTED]: vertexId=vertex_1555406634668_0016_2_00, startRequestedTime=1555415589721, startedTime=1555415589736 2019-04-16 11:53:09,736 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_2][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1555406634668_0016_2_00, reconfigureDoneTime=1555415589736, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@dfcb14d2, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true 2019-04-16 11:53:09,736 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1555406634668_0016_2_00 [Map 1] transitioned from INITED to RUNNING due to event V_START 2019-04-16 11:53:09,738 [INFO] [App Shared Pool - #2] |impl.RootInputVertexManager|: All source tasks completed. Ramping up 1 remaining tasks for vertex: Map 1 2019-04-16 11:53:09,738 [INFO] [App Shared Pool - #2] |impl.RootInputVertexManager|: Scheduling 1 tasks for vertex: Map 1 with totalTasks: 1. 0 source tasks completed out of 0. MinSourceTaskCompletedFraction: 1.0 min: 0.0 max: 0.0 2019-04-16 11:53:09,738 [INFO] [App Shared Pool - #2] |impl.VertexImpl|: Routing pending task events for vertex: vertex_1555406634668_0016_2_00 [Map 1] 2019-04-16 11:53:09,740 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_2][Event:TASK_STARTED]: vertexName=Map 1, taskId=task_1555406634668_0016_2_00_000000, scheduledTime=1555415589740, launchTime=1555415589740 2019-04-16 11:53:09,742 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1555406634668_0016_2_00_000000_0 with request: Capability[]Priority[2] host: ip-172-31-91-60.ec2.internal rack: null 2019-04-16 11:53:11,405 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_1555406634668_0016_01_000003, task=attempt_1555406634668_0016_2_00_000000_0, containerHost=ip-172-31-91-60.ec2.internal:8041, containerPriority= 2, containerResources=, localityMatchType=NodeLocal, matchedLocation=ip-172-31-91-60.ec2.internal, honorLocalityFlags=true, reusedContainer=false, delayedContainers=0 2019-04-16 11:53:11,408 [INFO] [ContainerLauncher #2] |launcher.TezContainerLauncherImpl|: Launching container_1555406634668_0016_01_000003 2019-04-16 11:53:11,417 [INFO] [ContainerLauncher #2] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1555406634668_0016_01_000003, launchTime=1555415591417 2019-04-16 11:53:12,512 [INFO] [Socket Reader #1 for port 46455] |ipc.Server|: Auth successful for application_1555406634668_0016 (auth:SIMPLE) 2019-04-16 11:53:12,523 [INFO] [Dispatcher thread {Central}] |impl.TaskAttemptImpl|: TaskAttempt: [attempt_1555406634668_0016_2_00_000000_0] submitted. Is using containerId: [container_1555406634668_0016_01_000003] on NM: [ip-172-31-91-60.ec2.internal:8041] 2019-04-16 11:53:12,524 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_2][Event:TASK_ATTEMPT_STARTED]: vertexName=Map 1, taskAttemptId=attempt_1555406634668_0016_2_00_000000_0, startTime=1555415592523, containerId=container_1555406634668_0016_01_000003, nodeId=ip-172-31-91-60.ec2.internal:8041 2019-04-16 11:53:12,621 [INFO] [IPC Server handler 5 on 46455] |impl.TaskImpl|: TaskAttempt:attempt_1555406634668_0016_2_00_000000_0 sent events: (0-1). 2019-04-16 11:53:19,759 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_2][Event:TASK_ATTEMPT_FINISHED]: vertexName=Map 1, taskAttemptId=attempt_1555406634668_0016_2_00_000000_0, creationTime=1555415589740, allocationTime=1555415591408, startTime=1555415592523, finishTime=1555415599759, timeTaken=7236, status=SUCCEEDED, diagnostics= 2019-04-16 11:53:19,759 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Task succeeded with attempt attempt_1555406634668_0016_2_00_000000_0 2019-04-16 11:53:19,761 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_2][Event:TASK_FINISHED]: vertexName=Map 1, taskId=task_1555406634668_0016_2_00_000000, startTime=1555415592523, finishTime=1555415599759, timeTaken=7236, status=SUCCEEDED, successfulAttemptID=attempt_1555406634668_0016_2_00_000000_0, diagnostics=, counters=Counters: 18, org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1, File System Counters, HDFS_BYTES_READ=2, HDFS_BYTES_WRITTEN=154, HDFS_READ_OPS=3, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=2, org.apache.tez.common.counters.TaskCounter, GC_TIME_MILLIS=65, CPU_MILLISECONDS=2900, PHYSICAL_MEMORY_BYTES=308281344, VIRTUAL_MEMORY_BYTES=4532211712, COMMITTED_HEAP_BYTES=308281344, INPUT_RECORDS_PROCESSED=1, INPUT_SPLIT_LENGTH_BYTES=2, OUTPUT_RECORDS=0, HIVE, CREATED_FILES=1, DESERIALIZE_ERRORS=0, RECORDS_IN_Map_1=1, RECORDS_OUT_0=1 2019-04-16 11:53:19,761 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Task Completion: vertex_1555406634668_0016_2_00 [Map 1], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null 2019-04-16 11:53:19,761 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: All tasks have succeeded, vertex:vertex_1555406634668_0016_2_00 [Map 1] 2019-04-16 11:53:19,762 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_2][Event:VERTEX_FINISHED]: vertexName=Map 1, vertexId=vertex_1555406634668_0016_2_00, initRequestedTime=1555415589697, initedTime=1555415589736, startRequestedTime=1555415589721, startedTime=1555415589736, finishTime=1555415599761, timeTaken=10025, status=SUCCEEDED, diagnostics=, counters=Counters: 18, org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1, File System Counters, HDFS_BYTES_READ=2, HDFS_BYTES_WRITTEN=154, HDFS_READ_OPS=3, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=2, org.apache.tez.common.counters.TaskCounter, GC_TIME_MILLIS=65, CPU_MILLISECONDS=2900, PHYSICAL_MEMORY_BYTES=308281344, VIRTUAL_MEMORY_BYTES=4532211712, COMMITTED_HEAP_BYTES=308281344, INPUT_RECORDS_PROCESSED=1, INPUT_SPLIT_LENGTH_BYTES=2, OUTPUT_RECORDS=0, HIVE, CREATED_FILES=1, DESERIALIZE_ERRORS=0, RECORDS_IN_Map_1=1, RECORDS_OUT_0=1, vertexStats=firstTaskStartTime=1555415592523, firstTasksToStart=[ task_1555406634668_0016_2_00_000000 ], lastTaskFinishTime=1555415599759, lastTasksToFinish=[ task_1555406634668_0016_2_00_000000 ], minTaskDuration=7236, maxTaskDuration=7236, avgTaskDuration=7236.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1555406634668_0016_2_00_000000 ], longestDurationTasks=[ task_1555406634668_0016_2_00_000000 ], vertexTaskStats={numFailedTaskAttempts=0, numKilledTaskAttempts=0, numCompletedTasks=1, numSucceededTasks=1, numKilledTasks=0, numFailedTasks=0}, servicePluginInfo=ServicePluginInfo {containerLauncherName=TezYarn, taskSchedulerName=TezYarn, taskCommunicatorName=TezYarn, containerLauncherClassName=org.apache.tez.dag.app.launcher.TezContainerLauncherImpl, taskSchedulerClassName=org.apache.tez.dag.app.rm.YarnTaskSchedulerService, taskCommunicatorClassName=org.apache.tez.dag.app.TezTaskCommunicatorImpl } 2019-04-16 11:53:19,762 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1555406634668_0016_2_00 [Map 1] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED 2019-04-16 11:53:19,762 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Vertex vertex_1555406634668_0016_2_00 [Map 1] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=1 2019-04-16 11:53:19,762 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=1, commitInProgress=0, terminationCause=null 2019-04-16 11:53:19,762 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: No exclusive output committers for vertex: vertex_1555406634668_0016_2_00 [Map 1] 2019-04-16 11:53:19,766 [INFO] [Dispatcher thread {Central}] |recovery.RecoveryService|: DAG completed, dagId=dag_1555406634668_0016_2, queueSize=0 2019-04-16 11:53:19,769 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_2][Event:DAG_FINISHED]: dagId=dag_1555406634668_0016_2, startTime=1555415589697, finishTime=1555415599762, timeTaken=10065, status=SUCCEEDED, diagnostics=, counters=Counters: 22, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=1, DATA_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=450, AM_GC_TIME_MILLIS=0, File System Counters, HDFS_BYTES_READ=2, HDFS_BYTES_WRITTEN=154, HDFS_READ_OPS=3, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=2, org.apache.tez.common.counters.TaskCounter, GC_TIME_MILLIS=65, CPU_MILLISECONDS=2900, PHYSICAL_MEMORY_BYTES=308281344, VIRTUAL_MEMORY_BYTES=4532211712, COMMITTED_HEAP_BYTES=308281344, INPUT_RECORDS_PROCESSED=1, INPUT_SPLIT_LENGTH_BYTES=2, OUTPUT_RECORDS=0, HIVE, CREATED_FILES=1, DESERIALIZE_ERRORS=0, RECORDS_IN_Map_1=1, RECORDS_OUT_0=1 2019-04-16 11:53:19,769 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: DAG: dag_1555406634668_0016_2 finished with state: SUCCEEDED 2019-04-16 11:53:19,769 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: dag_1555406634668_0016_2 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED 2019-04-16 11:53:19,769 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: DAG completed, dagId=dag_1555406634668_0016_2, dagState=SUCCEEDED 2019-04-16 11:53:19,769 [INFO] [Dispatcher thread {Central}] |common.TezUtilsInternal|: Redirecting log file based on addend: dag_1555406634668_0016_2_post End of LogType:syslog_dag_1555406634668_0016_2 LogType:syslog_dag_1555406634668_0016_2_post Log Upload Time:Tue Apr 16 11:57:49 +0000 2019 LogLength:8516 Log Contents: 2019-04-16 11:53:19,770 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Central Dispatcher queue size after DAG completion, before cleanup: 0 2019-04-16 11:53:19,770 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Waiting for next DAG to be submitted. 2019-04-16 11:53:19,770 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Cleaning up DAG: name=Select T.Candidate.startTime, T.Candidat...T(Stage-1), with id=dag_1555406634668_0016_2 2019-04-16 11:53:19,770 [INFO] [Dispatcher thread {Central}] |container.AMContainerMap|: Cleaned up completed containers on dagComplete. Removed=1, Remaining=1 2019-04-16 11:53:19,770 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Completed cleanup for DAG: name=Select T.Candidate.startTime, T.Candidat...T(Stage-1), with id=dag_1555406634668_0016_2 2019-04-16 11:53:26,422 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: Free: pendingRequests: 0 delayedContainers: 1 heartbeats: 101 lastPreemptionHeartbeat: 100 2019-04-16 11:53:28,763 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: No taskRequests. Container's idle timeout delay expired or is new. Releasing container, containerId=container_1555406634668_0016_01_000003, containerExpiryTime=1555415608543, idleTimeout=5000, taskRequestsCount=0, heldContainers=1, delayedContainers=0, isNew=false 2019-04-16 11:53:28,763 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1555406634668_0016_2][Event:CONTAINER_STOPPED]: containerId=container_1555406634668_0016_01_000003, stoppedTime=1555415608763, exitStatus=0 2019-04-16 11:53:28,764 [INFO] [ContainerLauncher #3] |launcher.TezContainerLauncherImpl|: Stopping container_1555406634668_0016_01_000003 2019-04-16 11:53:29,426 [INFO] [Dispatcher thread {Central}] |container.AMContainerImpl|: Container container_1555406634668_0016_01_000003 exited with diagnostics set to Container failed, exitCode=-105. Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2019-04-16 11:54:16,476 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: Free: pendingRequests: 0 delayedContainers: 0 heartbeats: 151 lastPreemptionHeartbeat: 150 2019-04-16 11:55:06,530 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: Free: pendingRequests: 0 delayedContainers: 0 heartbeats: 201 lastPreemptionHeartbeat: 200 2019-04-16 11:55:56,581 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: Free: pendingRequests: 0 delayedContainers: 0 heartbeats: 251 lastPreemptionHeartbeat: 250 2019-04-16 11:56:46,632 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: Free: pendingRequests: 0 delayedContainers: 0 heartbeats: 301 lastPreemptionHeartbeat: 300 2019-04-16 11:57:36,680 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: Free: pendingRequests: 0 delayedContainers: 0 heartbeats: 351 lastPreemptionHeartbeat: 350 2019-04-16 11:57:42,598 [INFO] [IPC Server handler 0 on 37007] |client.DAGClientHandler|: Received message to shutdown AM from hadoop (auth:SIMPLE) at 172.31.90.92 2019-04-16 11:57:42,598 [INFO] [IPC Server handler 0 on 37007] |rm.TaskSchedulerManager|: TaskScheduler notified that it should unregister from RM 2019-04-16 11:57:42,598 [INFO] [IPC Server handler 0 on 37007] |app.DAGAppMaster|: No current running DAG, shutting down the AM 2019-04-16 11:57:42,598 [INFO] [IPC Server handler 0 on 37007] |app.DAGAppMaster|: DAGAppMasterShutdownHandler invoked 2019-04-16 11:57:42,598 [INFO] [IPC Server handler 0 on 37007] |app.DAGAppMaster|: Handling DAGAppMaster shutdown 2019-04-16 11:57:42,599 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Sleeping for 5000 ms before shutting down 2019-04-16 11:57:47,599 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Calling stop for all the services 2019-04-16 11:57:47,599 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Initiating stop of YarnTaskScheduler 2019-04-16 11:57:47,599 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Releasing held containers 2019-04-16 11:57:47,599 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Removing all pending taskRequests 2019-04-16 11:57:47,600 [INFO] [AMShutdownThread] |history.HistoryEventHandler|: Stopping HistoryEventHandler 2019-04-16 11:57:47,600 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Stopping RecoveryService 2019-04-16 11:57:47,600 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Handle the remaining events in queue, queue size=0 2019-04-16 11:57:47,600 [INFO] [RecoveryEventHandlingThread] |recovery.RecoveryService|: EventQueue take interrupted. Returning 2019-04-16 11:57:47,600 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Closing Summary Stream 2019-04-16 11:57:47,606 [WARN] [AMShutdownThread] |recovery.RecoveryService|: Ignoring error while closing summary stream. The recovery directory at hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/.tez/application_1555406634668_0016/recovery/1 has already been deleted externally 2019-04-16 11:57:47,606 [INFO] [AMShutdownThread] |ats.ATSHistoryLoggingService|: Stopping ATSService, eventQueueBacklog=0 2019-04-16 11:57:47,612 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: AllocatedContainerManager Thread interrupted 2019-04-16 11:57:47,614 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Unregistering application from RM, exitStatus=SUCCEEDED, exitMessage=Session stats:submittedDAGs=2, successfulDAGs=2, failedDAGs=0, killedDAGs=0 , trackingURL=http://ip-172-31-90-92.ec2.internal:8080/tez-ui/#/tez-app/application_1555406634668_0016 2019-04-16 11:57:47,618 [INFO] [AMShutdownThread] |impl.AMRMClientImpl|: Waiting for application to be successfully unregistered. 2019-04-16 11:57:47,719 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Successfully unregistered application from RM 2019-04-16 11:57:47,719 [INFO] [AMShutdownThread] |rm.TaskSchedulerManager|: Shutting down AppCallbackExecutor 2019-04-16 11:57:47,719 [INFO] [AMRM Callback Handler Thread] |impl.AMRMClientAsyncImpl|: Interrupted while waiting for queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:323) 2019-04-16 11:57:47,721 [INFO] [AMShutdownThread] |mortbay.log|: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:0 2019-04-16 11:57:47,821 [INFO] [AMShutdownThread] |ipc.Server|: Stopping server on 46455 2019-04-16 11:57:47,822 [INFO] [AMShutdownThread] |ipc.Server|: Stopping server on 37007 2019-04-16 11:57:47,822 [INFO] [IPC Server Responder] |ipc.Server|: Stopping IPC Server Responder 2019-04-16 11:57:47,822 [INFO] [IPC Server listener on 46455] |ipc.Server|: Stopping IPC Server listener on 46455 2019-04-16 11:57:47,822 [INFO] [IPC Server listener on 37007] |ipc.Server|: Stopping IPC Server listener on 37007 2019-04-16 11:57:47,823 [INFO] [IPC Server Responder] |ipc.Server|: Stopping IPC Server Responder 2019-04-16 11:57:47,825 [INFO] [pool-1-thread-1] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked 2019-04-16 11:57:47,825 [INFO] [pool-1-thread-1] |app.DAGAppMaster|: The shutdown handler is still running, waiting for it to complete 2019-04-16 11:57:47,828 [WARN] [AMShutdownThread] |app.DAGAppMaster|: Failed to delete tez scratch data dir, path=hdfs://ip-172-31-90-92.ec2.internal:8020/tmp/hive/hadoop/_tez_session_dir/dcfadd60-b2ed-4a67-abea-7f1948953294/.tez/application_1555406634668_0016 2019-04-16 11:57:47,828 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Exiting DAGAppMaster..GoodBye! 2019-04-16 11:57:47,828 [INFO] [pool-1-thread-1] |app.DAGAppMaster|: The shutdown handler has completed End of LogType:syslog_dag_1555406634668_0016_2_post