Container: container_e703_1611750110647_64528_01_000002 on wx12-dsj-node008_45454_1611898349442 LogAggregationType: AGGREGATED =============================================================================================== LogType:container-localizer-syslog LogLastModifiedTime:Fri Jan 29 13:32:29 +0800 2021 LogLength:355 LogContents: 2021-01-29 13:32:15,461 WARN [main] org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer: Localization running as yarn not data_quality 2021-01-29 13:32:15,562 INFO [main] org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer: Disk Validator: yarn.nodemanager.disk-validator is loaded. End of LogType:container-localizer-syslog ******************************************************************************************* Container: container_e703_1611750110647_64528_01_000002 on wx12-dsj-node008_45454_1611898349442 LogAggregationType: AGGREGATED =============================================================================================== LogType:directory.info LogLastModifiedTime:Fri Jan 29 13:32:29 +0800 2021 LogLength:23116 LogContents: ls -l: total 12 -rw------- 1 yarn hadoop 114 Jan 29 13:32 container_tokens lrwxrwxrwx 1 yarn hadoop 158 Jan 29 13:32 hive-exec-3.1.0.3.1.5.0-152-52ac635d1c2eaf5d3e864f9c6dd1cc6c640a4133a03a6747e20ad2156829ebdf.jar -> /data01/hdp/yarn/local/usercache/data_quality/filecache/21259/hive-exec-3.1.0.3.1.5.0-152-52ac635d1c2eaf5d3e864f9c6dd1cc6c640a4133a03a6747e20ad2156829ebdf.jar lrwxrwxrwx 1 yarn hadoop 84 Jan 29 13:32 hive-hcatalog-core.jar -> /data01/hdp/yarn/local/usercache/data_quality/filecache/47184/hive-hcatalog-core.jar -rwx------ 1 yarn hadoop 5500 Jan 29 13:32 launch_container.sh lrwxrwxrwx 1 yarn hadoop 111 Jan 29 13:32 tez-conf.pb -> /data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/filecache/10/tez-conf.pb lrwxrwxrwx 1 yarn hadoop 47 Jan 29 13:32 tezlib -> /data01/hdp/yarn/local/filecache/771/tez.tar.gz drwxr-s--- 2 yarn hadoop 10 Jan 29 13:32 tmp find -L . -maxdepth 5 -ls: 7775705200 0 drwxr-s--- 3 yarn hadoop 267 Jan 29 13:32 . 92079512 0 drwxr-s--- 2 yarn hadoop 10 Jan 29 13:32 ./tmp 7773745123 8 -rwx------ 1 yarn hadoop 5500 Jan 29 13:32 ./launch_container.sh 7773745125 4 -rw------- 1 yarn hadoop 114 Jan 29 13:32 ./container_tokens 4354247297 264 -r-x------ 1 yarn hadoop 269276 Jan 29 13:32 ./hive-hcatalog-core.jar 6442452593 4 drwxr-xr-x 3 yarn hadoop 4096 May 28 2020 ./tezlib 6451152516 4 -r-xr-xr-x 1 yarn hadoop 1509 Dec 13 2019 ./tezlib/NOTICE 6451152517 4 -r-xr-xr-x 1 yarn hadoop 1045 Dec 13 2019 ./tezlib/LICENSE-MIT 6451152518 36 -r-xr-xr-x 1 yarn hadoop 36261 Dec 13 2019 ./tezlib/LICENSE-CDDLv1.1-GPLv2_withCPE 6451152519 20 -r-xr-xr-x 1 yarn hadoop 16792 Dec 13 2019 ./tezlib/LICENSE-CDDLv1.0 6451152520 4 -r-xr-xr-x 1 yarn hadoop 1465 Dec 13 2019 ./tezlib/LICENSE-BSD-3clause 6451152521 20 -r-xr-xr-x 1 yarn hadoop 17257 Dec 13 2019 ./tezlib/LICENSE 6451152522 8 -r-xr-xr-x 1 yarn hadoop 4128 Dec 13 2019 ./tezlib/LICENSE-SIL_OpenFontLicense-v1.1 6451152523 12 -r-xr-xr-x 1 yarn hadoop 8897 Dec 13 2019 ./tezlib/hadoop-shim-0.9.1.3.1.5.0-152.jar 6451152524 1020 -r-xr-xr-x 1 yarn hadoop 1042290 Dec 13 2019 ./tezlib/tez-api-0.9.1.3.1.5.0-152.jar 6451152529 84 -r-xr-xr-x 1 yarn hadoop 85945 Dec 13 2019 ./tezlib/tez-common-0.9.1.3.1.5.0-152.jar 6451152530 196 -r-xr-xr-x 1 yarn hadoop 199575 Dec 13 2019 ./tezlib/tez-runtime-internals-0.9.1.3.1.5.0-152.jar 6451152531 764 -r-xr-xr-x 1 yarn hadoop 780915 Dec 13 2019 ./tezlib/tez-runtime-library-0.9.1.3.1.5.0-152.jar 6451152532 292 -r-xr-xr-x 1 yarn hadoop 296388 Dec 13 2019 ./tezlib/tez-mapreduce-0.9.1.3.1.5.0-152.jar 6451152533 56 -r-xr-xr-x 1 yarn hadoop 56879 Dec 13 2019 ./tezlib/tez-examples-0.9.1.3.1.5.0-152.jar 6451152534 1380 -r-xr-xr-x 1 yarn hadoop 1410092 Dec 13 2019 ./tezlib/tez-dag-0.9.1.3.1.5.0-152.jar 6451152535 156 -r-xr-xr-x 1 yarn hadoop 158853 Dec 13 2019 ./tezlib/tez-tests-0.9.1.3.1.5.0-152.jar 6451152536 108 -r-xr-xr-x 1 yarn hadoop 107600 Dec 13 2019 ./tezlib/tez-ext-service-tests-0.9.1.3.1.5.0-152.jar 6451152541 121292 -r-xr-xr-x 1 yarn hadoop 124202048 Dec 13 2019 ./tezlib/tez-ui-0.9.1.3.1.5.0-152.war 6451457906 76 -r-xr-xr-x 1 yarn hadoop 75759 Dec 13 2019 ./tezlib/tez-protobuf-history-plugin-0.9.1.3.1.5.0-152.jar 6451457907 28 -r-xr-xr-x 1 yarn hadoop 28274 Dec 13 2019 ./tezlib/tez-yarn-timeline-history-0.9.1.3.1.5.0-152.jar 6451457908 8 -r-xr-xr-x 1 yarn hadoop 7839 Dec 13 2019 ./tezlib/tez-yarn-timeline-history-with-acls-0.9.1.3.1.5.0-152.jar 6451457909 12 -r-xr-xr-x 1 yarn hadoop 11564 Dec 13 2019 ./tezlib/tez-yarn-timeline-cache-plugin-0.9.1.3.1.5.0-152.jar 6451457910 24 -r-xr-xr-x 1 yarn hadoop 23692 Dec 13 2019 ./tezlib/tez-yarn-timeline-history-with-fs-0.9.1.3.1.5.0-152.jar 6451457911 80 -r-xr-xr-x 1 yarn hadoop 79219 Dec 13 2019 ./tezlib/tez-history-parser-0.9.1.3.1.5.0-152.jar 6451457912 80 -r-xr-xr-x 1 yarn hadoop 79108 Dec 13 2019 ./tezlib/tez-job-analyzer-0.9.1.3.1.5.0-152.jar 6451457913 16 -r-xr-xr-x 1 yarn hadoop 15352 Dec 13 2019 ./tezlib/tez-javadoc-tools-0.9.1.3.1.5.0-152.jar 6451457921 8 -r-xr-xr-x 1 yarn hadoop 6443 Dec 13 2019 ./tezlib/hadoop-shim-2.8-0.9.1.3.1.5.0-152.jar 2147519081 12 drwxr-xr-x 3 yarn hadoop 8192 May 28 2020 ./tezlib/lib 2147519082 32 -r-xr-xr-x 1 yarn hadoop 32119 Oct 22 2016 ./tezlib/lib/slf4j-api-1.7.10.jar 2147519083 3028 -r-xr-xr-x 1 yarn hadoop 3099479 Dec 13 2019 ./tezlib/lib/hadoop-yarn-api-3.1.1.3.1.5.0-152.jar 2147519084 280 -r-xr-xr-x 1 yarn hadoop 284220 Oct 22 2016 ./tezlib/lib/commons-lang-2.6.jar 2147519085 2684 -r-xr-xr-x 1 yarn hadoop 2746681 Dec 13 2019 ./tezlib/lib/guava-28.0-jre.jar 2147519086 8 -r-xr-xr-x 1 yarn hadoop 4617 Dec 13 2019 ./tezlib/lib/failureaccess-1.0.1.jar 2147519087 4 -r-xr-xr-x 1 yarn hadoop 2199 Dec 13 2019 ./tezlib/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar 2147519088 36 -r-xr-xr-x 1 yarn hadoop 33031 Dec 13 2019 ./tezlib/lib/jsr305-3.0.0.jar 2147519089 196 -r-xr-xr-x 1 yarn hadoop 200629 Dec 5 2019 ./tezlib/lib/checker-qual-2.8.1.jar 2147519090 16 -r-xr-xr-x 1 yarn hadoop 13166 Dec 13 2019 ./tezlib/lib/error_prone_annotations-2.3.2.jar 2147519091 12 -r-xr-xr-x 1 yarn hadoop 8781 Dec 13 2019 ./tezlib/lib/j2objc-annotations-1.3.jar 2147519092 4 -r-xr-xr-x 1 yarn hadoop 3448 Dec 5 2019 ./tezlib/lib/animal-sniffer-annotations-1.17.jar 2147519093 64 -r-xr-xr-x 1 yarn hadoop 62050 Oct 22 2016 ./tezlib/lib/commons-logging-1.1.3.jar 2147519094 100 -r-xr-xr-x 1 yarn hadoop 102244 Dec 5 2019 ./tezlib/lib/jaxb-api-2.2.11.jar 2147519095 524 -r-xr-xr-x 1 yarn hadoop 533455 Dec 13 2019 ./tezlib/lib/protobuf-java-2.5.0.jar 2147519096 68 -r-xr-xr-x 1 yarn hadoop 67889 Dec 6 2019 ./tezlib/lib/jackson-annotations-2.10.0.jar 2147519097 3956 -r-xr-xr-x 1 yarn hadoop 4050116 Dec 13 2019 ./tezlib/lib/hadoop-common-3.1.1.3.1.5.0-152.jar 2147519098 60 -r-xr-xr-x 1 yarn hadoop 59895 Dec 13 2019 ./tezlib/lib/hadoop-annotations-3.1.1.3.1.5.0-152.jar 2147519099 44 -r-xr-xr-x 1 yarn hadoop 41123 Oct 22 2016 ./tezlib/lib/commons-cli-1.2.jar 2147519100 1564 -r-xr-xr-x 1 yarn hadoop 1599627 Oct 22 2016 ./tezlib/lib/commons-math3-3.1.1.jar 2147519101 720 -r-xr-xr-x 1 yarn hadoop 736658 Oct 22 2016 ./tezlib/lib/httpclient-4.5.2.jar 2147519102 320 -r-xr-xr-x 1 yarn hadoop 326724 Oct 22 2016 ./tezlib/lib/httpcore-4.4.4.jar 2147519103 60 -r-xr-xr-x 1 yarn hadoop 58160 Oct 22 2016 ./tezlib/lib/commons-codec-1.4.jar 2147525666 184 -r-xr-xr-x 1 yarn hadoop 185140 Oct 22 2016 ./tezlib/lib/commons-io-2.4.jar 2147525683 304 -r-xr-xr-x 1 yarn hadoop 307410 Dec 5 2019 ./tezlib/lib/commons-net-3.6.jar 2147497318 576 -r-xr-xr-x 1 yarn hadoop 588337 Oct 22 2016 ./tezlib/lib/commons-collections-3.2.2.jar 2147497320 96 -r-xr-xr-x 1 yarn hadoop 95806 Oct 22 2016 ./tezlib/lib/javax.servlet-api-3.1.0.jar 2149096947 120 -r-xr-xr-x 1 yarn hadoop 118802 Dec 5 2019 ./tezlib/lib/jetty-servlet-9.3.24.v20180605.jar 2149096950 96 -r-xr-xr-x 1 yarn hadoop 95305 Dec 5 2019 ./tezlib/lib/jetty-security-9.3.24.v20180605.jar 2147519136 116 -r-xr-xr-x 1 yarn hadoop 114983 Dec 5 2019 ./tezlib/lib/jetty-webapp-9.3.24.v20180605.jar 2147519137 52 -r-xr-xr-x 1 yarn hadoop 51734 Dec 5 2019 ./tezlib/lib/jetty-xml-9.3.24.v20180605.jar 2147519138 128 -r-xr-xr-x 1 yarn hadoop 128719 Oct 22 2016 ./tezlib/lib/jersey-servlet-1.19.jar 2147519141 164 -r-xr-xr-x 1 yarn hadoop 165345 Oct 22 2016 ./tezlib/lib/jersey-json-1.19.jar 2147519142 80 -r-xr-xr-x 1 yarn hadoop 81743 Oct 22 2016 ./tezlib/lib/jettison-1.3.4.jar 2147519143 872 -r-xr-xr-x 1 yarn hadoop 890168 Oct 22 2016 ./tezlib/lib/jaxb-impl-2.2.3-1.jar 2147519150 224 -r-xr-xr-x 1 yarn hadoop 228286 Oct 22 2016 ./tezlib/lib/jackson-core-asl-1.9.2.jar 2147519151 748 -r-xr-xr-x 1 yarn hadoop 765648 Oct 22 2016 ./tezlib/lib/jackson-mapper-asl-1.9.2.jar 2147519154 20 -r-xr-xr-x 1 yarn hadoop 18323 Oct 22 2016 ./tezlib/lib/jackson-jaxrs-1.9.2.jar 2147519155 28 -r-xr-xr-x 1 yarn hadoop 27075 Oct 22 2016 ./tezlib/lib/jackson-xc-1.9.2.jar 2147519156 480 -r-xr-xr-x 1 yarn hadoop 489884 Oct 22 2016 ./tezlib/lib/log4j-1.2.17.jar 2147519157 244 -r-xr-xr-x 1 yarn hadoop 246918 Dec 6 2019 ./tezlib/lib/commons-beanutils-1.9.4.jar 2147519158 604 -r-xr-xr-x 1 yarn hadoop 616888 Dec 5 2019 ./tezlib/lib/commons-configuration2-2.1.1.jar 2147519159 428 -r-xr-xr-x 1 yarn hadoop 434678 Oct 22 2016 ./tezlib/lib/commons-lang3-3.4.jar 2147519160 12 -r-xr-xr-x 1 yarn hadoop 8866 Oct 22 2016 ./tezlib/lib/slf4j-log4j12-1.7.10.jar 2147519161 428 -r-xr-xr-x 1 yarn hadoop 436303 Oct 22 2016 ./tezlib/lib/avro-1.7.7.jar 2147519162 32 -r-xr-xr-x 1 yarn hadoop 29555 Oct 22 2016 ./tezlib/lib/paranamer-2.3.jar 2147519163 1224 -r-xr-xr-x 1 yarn hadoop 1251514 Oct 22 2016 ./tezlib/lib/snappy-java-1.0.5.jar 2147519164 236 -r-xr-xr-x 1 yarn hadoop 241367 Oct 22 2016 ./tezlib/lib/commons-compress-1.4.1.jar 2147519165 96 -r-xr-xr-x 1 yarn hadoop 94672 Oct 22 2016 ./tezlib/lib/xz-1.0.jar 2147519166 128 -r-xr-xr-x 1 yarn hadoop 128414 Dec 13 2019 ./tezlib/lib/re2j-1.1.jar 2147519167 188 -r-xr-xr-x 1 yarn hadoop 190432 Dec 13 2019 ./tezlib/lib/gson-2.2.4.jar 2147519168 136 -r-xr-xr-x 1 yarn hadoop 138292 Dec 13 2019 ./tezlib/lib/hadoop-auth-3.1.1.3.1.5.0-152.jar 2147519169 296 -r-xr-xr-x 1 yarn hadoop 299508 Dec 5 2019 ./tezlib/lib/nimbus-jose-jwt-4.41.1.jar 2147519170 8 -r-xr-xr-x 1 yarn hadoop 4722 Oct 22 2016 ./tezlib/lib/jcip-annotations-1.0-1.jar 2147519171 120 -r-xr-xr-x 1 yarn hadoop 120316 Dec 5 2019 ./tezlib/lib/json-smart-2.3.jar 2147519172 32 -r-xr-xr-x 1 yarn hadoop 30035 Dec 5 2019 ./tezlib/lib/accessors-smart-1.2.jar 2147519173 56 -r-xr-xr-x 1 yarn hadoop 53297 Oct 22 2016 ./tezlib/lib/asm-5.0.4.jar 2147519174 824 -r-xr-xr-x 1 yarn hadoop 839856 Dec 13 2019 ./tezlib/lib/zookeeper-3.4.6.3.1.5.0-152.jar 2147519175 1300 -r-xr-xr-x 1 yarn hadoop 1330394 Oct 22 2016 ./tezlib/lib/netty-3.10.5.Final.jar 2147519176 200 -r-xr-xr-x 1 yarn hadoop 201953 Dec 5 2019 ./tezlib/lib/curator-framework-2.12.0.jar 2147519177 2368 -r-xr-xr-x 1 yarn hadoop 2423138 Dec 5 2019 ./tezlib/lib/curator-client-2.12.0.jar 2147519180 20 -r-xr-xr-x 1 yarn hadoop 20409 Dec 5 2019 ./tezlib/lib/kerb-simplekdc-1.0.1.jar 2147519181 112 -r-xr-xr-x 1 yarn hadoop 113017 Dec 5 2019 ./tezlib/lib/kerb-client-1.0.1.jar 2147519182 32 -r-xr-xr-x 1 yarn hadoop 30674 Dec 5 2019 ./tezlib/lib/kerby-config-1.0.1.jar 2147519183 224 -r-xr-xr-x 1 yarn hadoop 226672 Dec 5 2019 ./tezlib/lib/kerb-core-1.0.1.jar 2147519184 200 -r-xr-xr-x 1 yarn hadoop 204650 Dec 5 2019 ./tezlib/lib/kerby-pkix-1.0.1.jar 2147519185 100 -r-xr-xr-x 1 yarn hadoop 102174 Dec 5 2019 ./tezlib/lib/kerby-asn1-1.0.1.jar 2147519186 40 -r-xr-xr-x 1 yarn hadoop 40554 Dec 5 2019 ./tezlib/lib/kerby-util-1.0.1.jar 2147519187 64 -r-xr-xr-x 1 yarn hadoop 65464 Dec 5 2019 ./tezlib/lib/kerb-common-1.0.1.jar 2147519188 116 -r-xr-xr-x 1 yarn hadoop 116120 Dec 5 2019 ./tezlib/lib/kerb-crypto-1.0.1.jar 2147519189 36 -r-xr-xr-x 1 yarn hadoop 36708 Dec 5 2019 ./tezlib/lib/kerb-util-1.0.1.jar 2147519190 20 -r-xr-xr-x 1 yarn hadoop 18763 Dec 5 2019 ./tezlib/lib/token-provider-1.0.1.jar 2147519191 80 -r-xr-xr-x 1 yarn hadoop 80980 Dec 5 2019 ./tezlib/lib/kerb-admin-1.0.1.jar 2147519192 84 -r-xr-xr-x 1 yarn hadoop 82756 Dec 5 2019 ./tezlib/lib/kerb-server-1.0.1.jar 2147519193 20 -r-xr-xr-x 1 yarn hadoop 20046 Dec 5 2019 ./tezlib/lib/kerb-identity-1.0.1.jar 2147519194 32 -r-xr-xr-x 1 yarn hadoop 29134 Dec 5 2019 ./tezlib/lib/kerby-xdr-1.0.1.jar 2147519195 280 -r-xr-xr-x 1 yarn hadoop 283598 Dec 5 2019 ./tezlib/lib/curator-recipes-2.12.0.jar 2147519196 1468 -r-xr-xr-x 1 yarn hadoop 1502280 Dec 5 2019 ./tezlib/lib/htrace-core4-4.1.0-incubating.jar 2147519198 88 -r-xr-xr-x 1 yarn hadoop 87325 Oct 22 2016 ./tezlib/lib/jline-0.9.94.jar 2147519199 1372 -r-xr-xr-x 1 yarn hadoop 1400944 Dec 6 2019 ./tezlib/lib/jackson-databind-2.10.0.jar 2147519204 344 -r-xr-xr-x 1 yarn hadoop 348635 Dec 6 2019 ./tezlib/lib/jackson-core-2.10.0.jar 2147519205 160 -r-xr-xr-x 1 yarn hadoop 161867 Oct 22 2016 ./tezlib/lib/stax2-api-3.1.4.jar 2147519206 504 -r-xr-xr-x 1 yarn hadoop 512742 Dec 5 2019 ./tezlib/lib/woodstox-core-5.0.3.jar 2147519207 464 -r-xr-xr-x 1 yarn hadoop 473316 Dec 13 2019 ./tezlib/lib/hadoop-aws-3.1.1.3.1.5.0-152.jar 2147519208 96420 -r-xr-xr-x 1 yarn hadoop 98732349 Dec 6 2019 ./tezlib/lib/aws-java-sdk-bundle-1.11.375.jar 2147519139 488 -r-xr-xr-x 1 yarn hadoop 497322 Dec 13 2019 ./tezlib/lib/hadoop-azure-3.1.1.3.1.5.0-152.jar 2147519140 796 -r-xr-xr-x 1 yarn hadoop 812977 Dec 6 2019 ./tezlib/lib/azure-storage-7.0.0.jar 2147519144 12 -r-xr-xr-x 1 yarn hadoop 10288 Dec 6 2019 ./tezlib/lib/azure-keyvault-core-1.0.0.jar 2147519145 40 -r-xr-xr-x 1 yarn hadoop 38526 Dec 5 2019 ./tezlib/lib/jetty-util-ajax-9.3.24.v20180605.jar 2147519146 448 -r-xr-xr-x 1 yarn hadoop 458642 Dec 5 2019 ./tezlib/lib/jetty-util-9.3.24.v20180605.jar 2147519147 416 -r-xr-xr-x 1 yarn hadoop 424076 Dec 6 2019 ./tezlib/lib/wildfly-openssl-1.0.7.Final.jar 2147519148 68 -r-xr-xr-x 1 yarn hadoop 67086 Dec 13 2019 ./tezlib/lib/hadoop-azure-datalake-3.1.1.3.1.5.0-152.jar 2147519149 512 -r-xr-xr-x 1 yarn hadoop 520682 Dec 6 2019 ./tezlib/lib/azure-data-lake-store-sdk-2.3.3.jar 2147519152 4048 -r-xr-xr-x 1 yarn hadoop 4141576 Dec 13 2019 ./tezlib/lib/gcs-connector-hadoop3-1.9.17.3.1.5.0-152-shaded.jar 2147519153 8 -r-xr-xr-x 1 yarn hadoop 6849 Dec 13 2019 ./tezlib/lib/google-extensions-0.3.1.jar 2147518953 100 -r-xr-xr-x 1 yarn hadoop 100356 Dec 13 2019 ./tezlib/lib/flogger-0.3.1.jar 2147518973 16 -r-xr-xr-x 1 yarn hadoop 15388 Dec 13 2019 ./tezlib/lib/flogger-system-backend-0.3.1.jar 2147518974 8 -r-xr-xr-x 1 yarn hadoop 7586 Dec 13 2019 ./tezlib/lib/flogger-log4j-backend-0.3.1.jar 2147518975 28 -r-xr-xr-x 1 yarn hadoop 26514 Oct 22 2016 ./tezlib/lib/stax-api-1.0.1.jar 2147518976 2776 -r-xr-xr-x 1 yarn hadoop 2839848 Dec 13 2019 ./tezlib/lib/hadoop-yarn-common-3.1.1.3.1.5.0-152.jar 2147518977 428 -r-xr-xr-x 1 yarn hadoop 436689 Oct 22 2016 ./tezlib/lib/jersey-core-1.19.jar 2147518978 48 -r-xr-xr-x 1 yarn hadoop 46367 Oct 22 2016 ./tezlib/lib/jsr311-api-1.1.1.jar 2147518979 132 -r-xr-xr-x 1 yarn hadoop 134021 Oct 22 2016 ./tezlib/lib/jersey-client-1.19.jar 2147518980 76 -r-xr-xr-x 1 yarn hadoop 76983 Dec 13 2019 ./tezlib/lib/guice-servlet-4.0.jar 2147518981 656 -r-xr-xr-x 1 yarn hadoop 668235 Dec 13 2019 ./tezlib/lib/guice-4.0.jar 2147518982 4 -r-xr-xr-x 1 yarn hadoop 2497 Oct 22 2016 ./tezlib/lib/javax.inject-1.jar 2147518984 8 -r-xr-xr-x 1 yarn hadoop 4467 Oct 22 2016 ./tezlib/lib/aopalliance-1.0.jar 2147518985 16 -r-xr-xr-x 1 yarn hadoop 16151 Oct 22 2016 ./tezlib/lib/jersey-guice-1.19.jar 2147518986 36 -r-xr-xr-x 1 yarn hadoop 34991 Dec 6 2019 ./tezlib/lib/jackson-module-jaxb-annotations-2.10.0.jar 2147518987 116 -r-xr-xr-x 1 yarn hadoop 115498 Dec 6 2019 ./tezlib/lib/jakarta.xml.bind-api-2.3.2.jar 2147518988 44 -r-xr-xr-x 1 yarn hadoop 44399 Dec 6 2019 ./tezlib/lib/jakarta.activation-api-1.2.1.jar 2147518989 16 -r-xr-xr-x 1 yarn hadoop 15529 Dec 6 2019 ./tezlib/lib/jackson-jaxrs-json-provider-2.10.0.jar 2147518990 36 -r-xr-xr-x 1 yarn hadoop 33176 Dec 6 2019 ./tezlib/lib/jackson-jaxrs-base-2.10.0.jar 2147518991 300 -r-xr-xr-x 1 yarn hadoop 303191 Dec 13 2019 ./tezlib/lib/hadoop-yarn-client-3.1.1.3.1.5.0-152.jar 2147518992 736 -r-xr-xr-x 1 yarn hadoop 751238 Oct 22 2016 ./tezlib/lib/commons-collections4-4.1.jar 2147518993 512 -r-xr-xr-x 1 yarn hadoop 520813 Dec 5 2019 ./tezlib/lib/jetty-server-9.3.24.v20180605.jar 2147518994 148 -r-xr-xr-x 1 yarn hadoop 150638 Dec 5 2019 ./tezlib/lib/jetty-http-9.3.24.v20180605.jar 2147519221 120 -r-xr-xr-x 1 yarn hadoop 122669 Dec 5 2019 ./tezlib/lib/jetty-io-9.3.24.v20180605.jar 2147519222 24 -r-xr-xr-x 1 yarn hadoop 24239 Oct 22 2016 ./tezlib/lib/commons-daemon-1.0.13.jar 2147519223 1024 -r-xr-xr-x 1 yarn hadoop 1045744 Oct 22 2016 ./tezlib/lib/leveldbjni-all-1.8.jar 2147519224 4872 -r-xr-xr-x 1 yarn hadoop 4986447 Dec 13 2019 ./tezlib/lib/hadoop-hdfs-client-3.1.1.3.1.5.0-152.jar 2147519227 324 -r-xr-xr-x 1 yarn hadoop 331034 Dec 5 2019 ./tezlib/lib/okhttp-2.7.5.jar 2147519228 68 -r-xr-xr-x 1 yarn hadoop 65928 Dec 5 2019 ./tezlib/lib/okio-1.6.0.jar 2147519231 124 -r-xr-xr-x 1 yarn hadoop 124846 Oct 22 2016 ./tezlib/lib/RoaringBitmap-0.4.9.jar 2147519232 728 -r-xr-xr-x 1 yarn hadoop 745264 Dec 6 2019 ./tezlib/lib/async-http-client-1.9.40.jar 2147519234 1620 -r-xr-xr-x 1 yarn hadoop 1654823 Dec 13 2019 ./tezlib/lib/hadoop-mapreduce-client-core-3.1.1.3.1.5.0-152.jar 2147519235 788 -r-xr-xr-x 1 yarn hadoop 803565 Dec 13 2019 ./tezlib/lib/hadoop-mapreduce-client-common-3.1.1.3.1.5.0-152.jar 2147519236 80 -r-xr-xr-x 1 yarn hadoop 80533 Dec 13 2019 ./tezlib/lib/hadoop-yarn-server-web-proxy-3.1.1.3.1.5.0-152.jar 2147519237 1308 -r-xr-xr-x 1 yarn hadoop 1336863 Dec 13 2019 ./tezlib/lib/hadoop-yarn-server-common-3.1.1.3.1.5.0-152.jar 2147519238 224 -r-xr-xr-x 1 yarn hadoop 225702 Dec 13 2019 ./tezlib/lib/hadoop-yarn-registry-3.1.1.3.1.5.0-152.jar 2147519239 304 -r-xr-xr-x 1 yarn hadoop 307637 Oct 23 2016 ./tezlib/lib/dnsjava-2.1.7.jar 2147519240 56 -r-xr-xr-x 1 yarn hadoop 55236 Dec 5 2019 ./tezlib/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar 2147519241 1688 -r-xr-xr-x 1 yarn hadoop 1726527 Dec 5 2019 ./tezlib/lib/ehcache-3.3.1.jar 2147519242 132 -r-xr-xr-x 1 yarn hadoop 134308 Dec 5 2019 ./tezlib/lib/HikariCP-java7-2.4.12.jar 2147519243 776 -r-xr-xr-x 1 yarn hadoop 792442 Dec 5 2019 ./tezlib/lib/mssql-jdbc-6.2.1.jre7.jar 2147519244 4092 -r-xr-xr-x 1 yarn hadoop 4189874 Dec 6 2019 ./tezlib/lib/bcprov-jdk15on-1.60.jar 2147519245 780 -r-xr-xr-x 1 yarn hadoop 796532 Dec 6 2019 ./tezlib/lib/bcpkix-jdk15on-1.60.jar 2147519246 104 -r-xr-xr-x 1 yarn hadoop 105112 Oct 22 2016 ./tezlib/lib/servlet-api-2.5.jar 2147519247 112 -r-xr-xr-x 1 yarn hadoop 111908 Oct 22 2016 ./tezlib/lib/metrics-core-3.1.0.jar 2147519248 288 -r-xr-xr-x 1 yarn hadoop 294844 Dec 13 2019 ./tezlib/lib/hadoop-yarn-server-applicationhistoryservice-3.1.1.3.1.5.0-152.jar 2147519249 28 -r-xr-xr-x 1 yarn hadoop 28569 Oct 22 2016 ./tezlib/lib/objenesis-1.0.jar 2147519250 380 -r-xr-xr-x 1 yarn hadoop 387689 Dec 5 2019 ./tezlib/lib/fst-2.50.jar 2147519251 60 -r-xr-xr-x 1 yarn hadoop 58487 Dec 5 2019 ./tezlib/lib/java-util-1.9.0.jar 2147519252 76 -r-xr-xr-x 1 yarn hadoop 75232 Dec 5 2019 ./tezlib/lib/json-io-2.5.1.jar 2147519253 92 -r-xr-xr-x 1 yarn hadoop 93196 Dec 13 2019 ./tezlib/lib/hadoop-yarn-server-timeline-pluginstorage-3.1.1.3.1.5.0-152.jar 2147519254 4 drwxr-xr-x 3 yarn hadoop 4096 Dec 13 2019 ./tezlib/lib/native 2147519255 1784 -r-xr-xr-x 1 yarn hadoop 1824978 Dec 13 2019 ./tezlib/lib/native/libhadoop.a 2147519256 3324 -r-xr-xr-x 2 yarn hadoop 3402313 Dec 13 2019 ./tezlib/lib/native/libnativetask.so.1.0.0 2147519257 24 -r-xr-xr-x 3 yarn hadoop 23800 Jun 10 2014 ./tezlib/lib/native/libsnappy.so.1.1.4 2147519258 8396 -r-xr-xr-x 1 yarn hadoop 8594282 Dec 13 2019 ./tezlib/lib/native/libnativetask.a 2147519257 24 -r-xr-xr-x 3 yarn hadoop 23800 Jun 10 2014 ./tezlib/lib/native/libsnappy.so.1 2147519261 1008 -r-xr-xr-x 2 yarn hadoop 1029214 Dec 13 2019 ./tezlib/lib/native/libhadoop.so.1.0.0 2147519263 452 -r-xr-xr-x 1 yarn hadoop 459400 Dec 13 2019 ./tezlib/lib/native/libhdfs.a 2147518880 280 -r-xr-xr-x 2 yarn hadoop 286676 Dec 13 2019 ./tezlib/lib/native/libhdfs.so 2147519261 1008 -r-xr-xr-x 2 yarn hadoop 1029214 Dec 13 2019 ./tezlib/lib/native/libhadoop.so 2147519257 24 -r-xr-xr-x 3 yarn hadoop 23800 Jun 10 2014 ./tezlib/lib/native/libsnappy.so 2147518881 468 -r-xr-xr-x 1 yarn hadoop 476136 Dec 13 2019 ./tezlib/lib/native/libhadooputils.a 2147518882 0 drwxr-xr-x 2 yarn hadoop 94 Dec 13 2019 ./tezlib/lib/native/examples 2147518883 904 -r-xr-xr-x 1 yarn hadoop 924089 Dec 13 2019 ./tezlib/lib/native/examples/pipes-sort 2147518884 924 -r-xr-xr-x 1 yarn hadoop 943052 Dec 13 2019 ./tezlib/lib/native/examples/wordcount-part 2147518885 920 -r-xr-xr-x 1 yarn hadoop 938380 Dec 13 2019 ./tezlib/lib/native/examples/wordcount-simple 2147518886 964 -r-xr-xr-x 1 yarn hadoop 983078 Dec 13 2019 ./tezlib/lib/native/examples/wordcount-nopipe 2147518887 1572 -r-xr-xr-x 1 yarn hadoop 1607384 Dec 13 2019 ./tezlib/lib/native/libhadooppipes.a 2147518880 280 -r-xr-xr-x 2 yarn hadoop 286676 Dec 13 2019 ./tezlib/lib/native/libhdfs.so.0.0.0 2147519256 3324 -r-xr-xr-x 2 yarn hadoop 3402313 Dec 13 2019 ./tezlib/lib/native/libnativetask.so 74560820 128 -r-x------ 1 yarn hadoop 127507 Jan 29 13:32 ./tez-conf.pb 7776483309 41784 -r-x------ 1 yarn hadoop 42785708 Dec 4 15:33 ./hive-exec-3.1.0.3.1.5.0-152-52ac635d1c2eaf5d3e864f9c6dd1cc6c640a4133a03a6747e20ad2156829ebdf.jar broken symlinks(find -L . -maxdepth 5 -type l -ls): End of LogType:directory.info ******************************************************************************* Container: container_e703_1611750110647_64528_01_000002 on wx12-dsj-node008_45454_1611898349442 LogAggregationType: AGGREGATED =============================================================================================== LogType:launch_container.sh LogLastModifiedTime:Fri Jan 29 13:32:29 +0800 2021 LogLength:5500 LogContents: #!/bin/bash set -o pipefail -e export PRELAUNCH_OUT="/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/prelaunch.out" exec >"${PRELAUNCH_OUT}" export PRELAUNCH_ERR="/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/prelaunch.err" exec 2>"${PRELAUNCH_ERR}" echo "Setting up env variables" export JAVA_HOME=${JAVA_HOME:-"/usr/jdk64/jdk1.8.0_112"} export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/hdp/3.1.5.0-152/hadoop/conf"} export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/hdp/3.1.5.0-152/hadoop-yarn"} export HADOOP_HOME=${HADOOP_HOME:-"/usr/hdp/3.1.5.0-152/hadoop"} export PATH=${PATH:-"/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent"} export LANG=${LANG:-"en_US.UTF-8"} export HADOOP_TOKEN_FILE_LOCATION="/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/container_tokens" export CONTAINER_ID="container_e703_1611750110647_64528_01_000002" export NM_PORT="45454" export NM_HOST="wx12-dsj-node008" export NM_HTTP_PORT="8042" export LOCAL_DIRS="/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528" export LOCAL_USER_DIRS="/data01/hdp/yarn/local/usercache/data_quality/" export LOG_DIRS="/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002" export USER="data_quality" export LOGNAME="data_quality" export HOME="/home/" export PWD="/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002" export JVM_PID="$$" export MALLOC_ARENA_MAX="4" export NM_AUX_SERVICE_timeline_collector="" export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=" export NM_AUX_SERVICE_spark2_shuffle="" export SHELL="/bin/bash" export HADOOP_ROOT_LOGGER="INFO,CLA" export CLASSPATH="/usr/hdp/3.1.5.0-152/hadoop/lib/hadoop-lzo-0.6.0.3.1.5.0-152.jar:/etc/hadoop/conf/secure:$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*:" export LD_LIBRARY_PATH="$PWD:/usr/hdp/3.1.5.0-152/hadoop/lib/native:/usr/hdp/3.1.5.0-152/hadoop/lib/native/Linux-amd64-64:$PWD:/usr/hdp/3.1.5.0-152/hadoop/lib/native:/usr/hdp/3.1.5.0-152/hadoop/lib/native/Linux-amd64-64:$PWD:/usr/hdp/3.1.5.0-152/hadoop/lib/native:/usr/hdp/3.1.5.0-152/hadoop/lib/native/Linux-amd64-64:$PWD:/usr/hdp/3.1.5.0-152/hadoop/lib/native:/usr/hdp/3.1.5.0-152/hadoop/lib/native/Linux-amd64-64:$HADOOP_COMMON_HOME/lib/native/" echo "Setting up job resources" ln -sf "/data01/hdp/yarn/local/usercache/data_quality/filecache/47184/hive-hcatalog-core.jar" "hive-hcatalog-core.jar" ln -sf "/data01/hdp/yarn/local/filecache/771/tez.tar.gz" "tezlib" ln -sf "/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/filecache/10/tez-conf.pb" "tez-conf.pb" ln -sf "/data01/hdp/yarn/local/usercache/data_quality/filecache/21259/hive-exec-3.1.0.3.1.5.0-152-52ac635d1c2eaf5d3e864f9c6dd1cc6c640a4133a03a6747e20ad2156829ebdf.jar" "hive-exec-3.1.0.3.1.5.0-152-52ac635d1c2eaf5d3e864f9c6dd1cc6c640a4133a03a6747e20ad2156829ebdf.jar" echo "Copying debugging information" # Creating copy of launch script cp "launch_container.sh" "/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/launch_container.sh" chmod 640 "/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/launch_container.sh" # Determining directory contents echo "ls -l:" 1>"/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/directory.info" ls -l 1>>"/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/directory.info" echo "find -L . -maxdepth 5 -ls:" 1>>"/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/directory.info" find -L . -maxdepth 5 -ls 1>>"/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/directory.info" echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/directory.info" find -L . -maxdepth 5 -type l -ls 1>>"/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/directory.info" echo "Launching container" exec /bin/bash -c "$JAVA_HOME/bin/java -Xmx5734m -server -Djava.net.preferIPv4Stack=true -Dhdp.version=3.1.5.0-152 -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002 -Dtez.root.logger=INFO,CLA -Djava.io.tmpdir=$PWD/tmp org.apache.tez.runtime.task.TezChild wx12-dsj-node045 27287 container_e703_1611750110647_64528_01_000002 application_1611750110647_64528 1 1>/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/stdout 2>/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/stderr " End of LogType:launch_container.sh ************************************************************************************ End of LogType:prelaunch.err ****************************************************************************** Container: container_e703_1611750110647_64528_01_000002 on wx12-dsj-node008_45454_1611898349442 LogAggregationType: AGGREGATED =============================================================================================== LogType:prelaunch.out LogLastModifiedTime:Fri Jan 29 13:32:29 +0800 2021 LogLength:100 LogContents: Setting up env variables Setting up job resources Copying debugging information Launching container End of LogType:prelaunch.out ****************************************************************************** Container: container_e703_1611750110647_64528_01_000002 on wx12-dsj-node008_45454_1611898349442 LogAggregationType: AGGREGATED =============================================================================================== LogType:stderr LogLastModifiedTime:Fri Jan 29 13:32:29 +0800 2021 LogLength:380 LogContents: 2021-01-29 13:32:18 Starting to run new task attempt: attempt_1611750110647_64528_1_00_000000_0 2021-01-29 13:32:19 Completed running task attempt: attempt_1611750110647_64528_1_00_000000_0 2021-01-29 13:32:19 Starting to run new task attempt: attempt_1611750110647_64528_1_01_000000_0 2021-01-29 13:32:20 Completed running task attempt: attempt_1611750110647_64528_1_01_000000_0 End of LogType:stderr *********************************************************************** Container: container_e703_1611750110647_64528_01_000002 on wx12-dsj-node008_45454_1611898349442 LogAggregationType: AGGREGATED =============================================================================================== LogType:stdout LogLastModifiedTime:Fri Jan 29 13:32:29 +0800 2021 LogLength:4737 LogContents: 2021-01-29 13:32:18 Starting to run new task attempt: attempt_1611750110647_64528_1_00_000000_0 1.753: [GC pause (Metadata GC Threshold) (young) (initial-mark), 0.0660978 secs] [Parallel Time: 28.8 ms, GC Workers: 43] [GC Worker Start (ms): Min: 1753.2, Avg: 1755.5, Max: 1756.2, Diff: 3.0] [Ext Root Scanning (ms): Min: 1.5, Avg: 12.4, Max: 24.5, Diff: 23.1, Sum: 533.4] [Update RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] [Processed Buffers: Min: 0, Avg: 0.0, Max: 0, Diff: 0, Sum: 0] [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.5] [Code Root Scanning (ms): Min: 0.0, Avg: 1.1, Max: 10.4, Diff: 10.4, Sum: 46.4] [Object Copy (ms): Min: 0.0, Avg: 9.4, Max: 21.0, Diff: 21.0, Sum: 404.1] [Termination (ms): Min: 0.1, Avg: 2.6, Max: 5.0, Diff: 5.0, Sum: 109.8] [Termination Attempts: Min: 1, Avg: 90.2, Max: 187, Diff: 186, Sum: 3878] [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 1.1] [GC Worker Total (ms): Min: 24.8, Avg: 25.5, Max: 27.7, Diff: 2.9, Sum: 1095.4] [GC Worker End (ms): Min: 1781.0, Avg: 1781.0, Max: 1781.1, Diff: 0.1] [Code Root Fixup: 2.9 ms] [Code Root Purge: 0.1 ms] [Clear CT: 1.0 ms] [Other: 33.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 31.4 ms] [Ref Enq: 0.1 ms] [Redirty Cards: 0.7 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 0.4 ms] [Eden: 118.0M(227.0M)->0.0B(206.0M) Survivors: 0.0B->21.0M Heap: 118.0M(2048.0M)->20.2M(2048.0M)] [Times: user=0.85 sys=0.02, real=0.07 secs] 1.819: [GC concurrent-root-region-scan-start] 1.823: [GC concurrent-root-region-scan-end, 0.0043458 secs] 1.824: [GC concurrent-mark-start] 1.826: [GC concurrent-mark-end, 0.0021866 secs] 1.826: [GC remark 1.826: [Finalize Marking, 0.0059708 secs] 1.832: [GC ref-proc, 0.0012632 secs] 1.833: [Unloading, 0.0063138 secs], 0.0154943 secs] [Times: user=0.23 sys=0.02, real=0.01 secs] 1.842: [GC cleanup 21M->21M(2048M), 0.0083512 secs] [Times: user=0.10 sys=0.00, real=0.01 secs] 2021-01-29 13:32:19 Completed running task attempt: attempt_1611750110647_64528_1_00_000000_0 2021-01-29 13:32:19 Starting to run new task attempt: attempt_1611750110647_64528_1_01_000000_0 3.285: [GC pause (Metadata GC Threshold) (young) (initial-mark), 0.0657299 secs] [Parallel Time: 31.7 ms, GC Workers: 43] [GC Worker Start (ms): Min: 3288.3, Avg: 3289.5, Max: 3290.7, Diff: 2.4] [Ext Root Scanning (ms): Min: 2.7, Avg: 7.7, Max: 29.2, Diff: 26.6, Sum: 329.8] [Update RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] [Processed Buffers: Min: 0, Avg: 0.0, Max: 0, Diff: 0, Sum: 0] [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 1.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.2, Max: 5.5, Diff: 5.5, Sum: 7.0] [Object Copy (ms): Min: 0.0, Avg: 14.6, Max: 26.7, Diff: 26.7, Sum: 628.4] [Termination (ms): Min: 0.0, Avg: 7.4, Max: 8.1, Diff: 8.1, Sum: 317.5] [Termination Attempts: Min: 1, Avg: 28.3, Max: 58, Diff: 57, Sum: 1217] [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 1.2] [GC Worker Total (ms): Min: 28.7, Avg: 29.9, Max: 31.2, Diff: 2.5, Sum: 1285.0] [GC Worker End (ms): Min: 3319.4, Avg: 3319.4, Max: 3319.8, Diff: 0.4] [Code Root Fixup: 0.7 ms] [Code Root Purge: 0.0 ms] [Clear CT: 1.9 ms] [Other: 31.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 23.6 ms] [Ref Enq: 0.2 ms] [Redirty Cards: 1.5 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.5 ms] [Free CSet: 1.1 ms] [Eden: 183.0M(206.0M)->0.0B(198.0M) Survivors: 21.0M->29.0M Heap: 203.2M(2048.0M)->41.1M(2048.0M)] [Times: user=1.02 sys=0.03, real=0.07 secs] 3.351: [GC concurrent-root-region-scan-start] 3.358: [GC concurrent-root-region-scan-end, 0.0071657 secs] 3.358: [GC concurrent-mark-start] 3.362: [GC concurrent-mark-end, 0.0037682 secs] 3.362: [GC remark 3.362: [Finalize Marking, 0.0057564 secs] 3.368: [GC ref-proc, 0.0010711 secs] 3.369: [Unloading, 0.0076912 secs], 0.0169542 secs] [Times: user=0.22 sys=0.00, real=0.02 secs] 3.379: [GC cleanup 42M->42M(2048M), 0.0092725 secs] [Times: user=0.07 sys=0.01, real=0.01 secs] 2021-01-29 13:32:20 Completed running task attempt: attempt_1611750110647_64528_1_01_000000_0 Heap garbage-first heap total 2097152K, used 58507K [0x0000000659a00000, 0x0000000659b04000, 0x00000007c0000000) region size 1024K, 46 young (47104K), 29 survivors (29696K) Metaspace used 36678K, capacity 36982K, committed 37452K, reserved 1083392K class space used 4273K, capacity 4377K, committed 4428K, reserved 1048576K End of LogType:stdout *********************************************************************** Container: container_e703_1611750110647_64528_01_000002 on wx12-dsj-node008_45454_1611898349442 LogAggregationType: AGGREGATED =============================================================================================== LogType:syslog LogLastModifiedTime:Fri Jan 29 13:32:29 +0800 2021 LogLength:31881 LogContents: 2021-01-29 13:32:17,445 [INFO] [main] |task.TezChild|: TezChild starting with PID=56935, containerIdentifier=container_e703_1611750110647_64528_01_000002 2021-01-29 13:32:17,779 [INFO] [main] |shim.HadoopShimsLoader|: Trying to locate HadoopShimProvider for hadoopVersion=3.1.1.3.1.5.0-152, majorVersion=3, minorVersion=1 2021-01-29 13:32:17,781 [INFO] [main] |shim.HadoopShimsLoader|: Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=3.1.1.3.1.5.0-152, majorVersion=3, minorVersion=1 2021-01-29 13:32:17,812 [INFO] [main] |task.TezChild|: /************************************************************ [system properties] os.name: Linux os.version: 3.10.0-1127.8.2.el7.x86_64 java.home: /usr/jdk64/jdk1.8.0_112/jre java.runtime.version: 1.8.0_112-b15 java.vendor: Oracle Corporation java.version: 1.8.0_112 java.vm.name: Java HotSpot(TM) 64-Bit Server VM java.class.path: /usr/hdp/3.1.5.0-152/hadoop/lib/hadoop-lzo-0.6.0.3.1.5.0-152.jar:/etc/hadoop/conf/secure:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/hive-hcatalog-core.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/hive-exec-3.1.0.3.1.5.0-152-52ac635d1c2eaf5d3e864f9c6dd1cc6c640a4133a03a6747e20ad2156829ebdf.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/hadoop-shim-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-api-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-common-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-runtime-internals-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-runtime-library-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-mapreduce-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-examples-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-dag-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-tests-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-ext-service-tests-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-protobuf-history-plugin-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-yarn-timeline-history-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-yarn-timeline-history-with-acls-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-yarn-timeline-cache-plugin-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-yarn-timeline-history-with-fs-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-history-parser-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-job-analyzer-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/tez-javadoc-tools-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/hadoop-shim-2.8-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/slf4j-api-1.7.10.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-yarn-api-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/commons-lang-2.6.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/guava-28.0-jre.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/failureaccess-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jsr305-3.0.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/checker-qual-2.8.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/error_prone_annotations-2.3.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/j2objc-annotations-1.3.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/animal-sniffer-annotations-1.17.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/commons-logging-1.1.3.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jaxb-api-2.2.11.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/protobuf-java-2.5.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jackson-annotations-2.10.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-common-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-annotations-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/commons-cli-1.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/commons-math3-3.1.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/httpclient-4.5.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/httpcore-4.4.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/commons-codec-1.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/commons-io-2.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/commons-net-3.6.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/commons-collections-3.2.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/javax.servlet-api-3.1.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jetty-servlet-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jetty-security-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jetty-webapp-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jetty-xml-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jersey-servlet-1.19.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jersey-json-1.19.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jettison-1.3.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jaxb-impl-2.2.3-1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jackson-core-asl-1.9.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jackson-mapper-asl-1.9.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jackson-jaxrs-1.9.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jackson-xc-1.9.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/log4j-1.2.17.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/commons-beanutils-1.9.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/commons-configuration2-2.1.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/commons-lang3-3.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/slf4j-log4j12-1.7.10.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/avro-1.7.7.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/paranamer-2.3.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/snappy-java-1.0.5.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/commons-compress-1.4.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/xz-1.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/re2j-1.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/gson-2.2.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-auth-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/nimbus-jose-jwt-4.41.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jcip-annotations-1.0-1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/json-smart-2.3.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/accessors-smart-1.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/asm-5.0.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/zookeeper-3.4.6.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/netty-3.10.5.Final.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/curator-framework-2.12.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/curator-client-2.12.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/kerb-simplekdc-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/kerb-client-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/kerby-config-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/kerb-core-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/kerby-pkix-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/kerby-asn1-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/kerby-util-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/kerb-common-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/kerb-crypto-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/kerb-util-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/token-provider-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/kerb-admin-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/kerb-server-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/kerb-identity-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/kerby-xdr-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/curator-recipes-2.12.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/htrace-core4-4.1.0-incubating.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jline-0.9.94.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jackson-databind-2.10.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jackson-core-2.10.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/stax2-api-3.1.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/woodstox-core-5.0.3.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-aws-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/aws-java-sdk-bundle-1.11.375.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-azure-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/azure-storage-7.0.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/azure-keyvault-core-1.0.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jetty-util-ajax-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jetty-util-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/wildfly-openssl-1.0.7.Final.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-azure-datalake-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/azure-data-lake-store-sdk-2.3.3.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/gcs-connector-hadoop3-1.9.17.3.1.5.0-152-shaded.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/google-extensions-0.3.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/flogger-0.3.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/flogger-system-backend-0.3.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/flogger-log4j-backend-0.3.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/stax-api-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-yarn-common-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jersey-core-1.19.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jsr311-api-1.1.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jersey-client-1.19.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/guice-servlet-4.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/guice-4.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/javax.inject-1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/aopalliance-1.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jersey-guice-1.19.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jackson-module-jaxb-annotations-2.10.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jakarta.xml.bind-api-2.3.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jakarta.activation-api-1.2.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jackson-jaxrs-json-provider-2.10.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jackson-jaxrs-base-2.10.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-yarn-client-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/commons-collections4-4.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jetty-server-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jetty-http-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/jetty-io-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/commons-daemon-1.0.13.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/leveldbjni-all-1.8.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-hdfs-client-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/okhttp-2.7.5.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/okio-1.6.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/RoaringBitmap-0.4.9.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/async-http-client-1.9.40.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-mapreduce-client-core-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-mapreduce-client-common-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-yarn-server-web-proxy-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-yarn-server-common-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-yarn-registry-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/dnsjava-2.1.7.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/ehcache-3.3.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/HikariCP-java7-2.4.12.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/mssql-jdbc-6.2.1.jre7.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/bcprov-jdk15on-1.60.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/bcpkix-jdk15on-1.60.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/servlet-api-2.5.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/metrics-core-3.1.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-yarn-server-applicationhistoryservice-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/objenesis-1.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/fst-2.50.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/java-util-1.9.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/json-io-2.5.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tezlib/lib/hadoop-yarn-server-timeline-pluginstorage-3.1.1.3.1.5.0-152.jar: java.io.tmpdir: /data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002/tmp user.dir: /data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000002 user.name: yarn ************************************************************/ 2021-01-29 13:32:17,815 [INFO] [main] |security.SecurityUtil|: Updating Configuration 2021-01-29 13:32:17,869 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e703_1611750110647_64528_01_000002 2021-01-29 13:32:18,176 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_e703_1611750110647_64528_01_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1611750110647_64528_1_00_000000_0 2021-01-29 13:32:18,178 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: attempt_1611750110647_64528_1_00_000000_0 2021-01-29 13:32:19,903 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e703_1611750110647_64528_01_000002 2021-01-29 13:32:19,973 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_e703_1611750110647_64528_01_000002: 68 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1611750110647_64528_1_01_000000_0 2021-01-29 13:32:19,974 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: attempt_1611750110647_64528_1_01_000000_0 2021-01-29 13:32:20,549 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e703_1611750110647_64528_01_000002 2021-01-29 13:32:25,996 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_e703_1611750110647_64528_01_000002: 5445 ms after starting to poll. TaskInfo: shouldDie: true 2021-01-29 13:32:25,996 [INFO] [main] |task.TezChild|: ContainerTask returned shouldDie=true for container container_e703_1611750110647_64528_01_000002, Exiting 2021-01-29 13:32:25,996 [INFO] [main] |task.TezChild|: Shutdown invoked for container container_e703_1611750110647_64528_01_000002 2021-01-29 13:32:25,996 [INFO] [main] |task.TezChild|: Shutting down container container_e703_1611750110647_64528_01_000002 End of LogType:syslog *********************************************************************** Container: container_e703_1611750110647_64528_01_000002 on wx12-dsj-node008_45454_1611898349442 LogAggregationType: AGGREGATED =============================================================================================== LogType:syslog_attempt_1611750110647_64528_1_00_000000_0 LogLastModifiedTime:Fri Jan 29 13:32:29 +0800 2021 LogLength:15500 LogContents: 2021-01-29 13:32:18,184 [INFO] [main] |task.TezChild|: Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0 2021-01-29 13:32:18,202 [INFO] [main] |metrics.TaskCounterUpdater|: Using ResourceCalculatorProcessTree : org.apache.tez.util.TezMxBeanResourceCalculator 2021-01-29 13:32:18,203 [INFO] [main] |runtime.LogicalIOProcessorRuntimeTask|: Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : select count(1) as counts, 0 a...'11' (Stage-1), VertexName: Map 1, VertexParallelism: 1, TaskAttemptID:attempt_1611750110647_64528_1_00_000000_0, processorName=org.apache.hadoop.hive.ql.exec.tez.MapTezProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=dm_common_helper_detect_click_last_trace_code_h, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInputLegacy }}, ], outputSpecList=[{{ destinationVertexName=Reducer 2, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.UnorderedPartitionedKVOutput }}, ], taskConfEntryCount=0 2021-01-29 13:32:18,204 [INFO] [main] |resources.MemoryDistributor|: InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=6012534784, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor 2021-01-29 13:32:18,210 [INFO] [TezChild] |task.TaskRunner2Callable|: Initializing task, taskAttemptId=attempt_1611750110647_64528_1_00_000000_0 2021-01-29 13:32:18,242 [INFO] [TezChild] |Configuration.deprecation|: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed 2021-01-29 13:32:18,243 [INFO] [TezChild] |Configuration.deprecation|: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive 2021-01-29 13:32:18,247 [INFO] [TaskHeartbeatThread] |task.TaskReporter|: Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1611750110647_64528_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0 2021-01-29 13:32:18,250 [INFO] [TezChild] |Configuration.deprecation|: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 2021-01-29 13:32:18,250 [INFO] [TezChild] |Configuration.deprecation|: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition 2021-01-29 13:32:18,250 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor 2021-01-29 13:32:18,250 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish 2021-01-29 13:32:18,263 [INFO] [I/O Setup 1 Initialize: {Reducer 2}] |writers.UnorderedPartitionedKVWriter|: Requested BufferSize (tez.runtime.unordered.output.buffer.size-mb) : 1 2021-01-29 13:32:18,263 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish 2021-01-29 13:32:18,290 [INFO] [I/O Setup 0 Initialize: {dm_common_helper_detect_click_last_trace_code_h}] |counters.Limits|: Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=3000, COUNTER_NAME_MAX=64, MAX_COUNTERS=10000 2021-01-29 13:32:18,297 [INFO] [I/O Setup 0 Initialize: {dm_common_helper_detect_click_last_trace_code_h}] |input.MRInput|: dm_common_helper_detect_click_last_trace_code_h using newmapreduce API=false, split via event=true, numPhysicalInputs=1 2021-01-29 13:32:18,297 [INFO] [I/O Setup 0 Initialize: {dm_common_helper_detect_click_last_trace_code_h}] |input.MRInputLegacy|: dm_common_helper_detect_click_last_trace_code_h MRInputLegacy deferring initialization 2021-01-29 13:32:18,298 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: All initializers finished 2021-01-29 13:32:18,301 [INFO] [TezChild] |resources.MemoryDistributor|: InitialRequests=[dm_common_helper_detect_click_last_trace_code_h:INPUT:0:org.apache.tez.mapreduce.input.MRInputLegacy], [Reducer 2:OUTPUT:1048576:org.apache.tez.runtime.library.output.UnorderedPartitionedKVOutput] 2021-01-29 13:32:18,307 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1] 2021-01-29 13:32:18,313 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996 2021-01-29 13:32:18,313 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: Scaling Requests. NumRequests: 2, numScaledRequests: 2, TotalRequested: 1048576, TotalRequestedScaled: 524288.0, TotalJVMHeap: 6012534784, TotalAvailable: 4028398305, TotalRequested/TotalJVMHeap:0.00 2021-01-29 13:32:18,314 [INFO] [TezChild] |resources.MemoryDistributor|: Allocations=[dm_common_helper_detect_click_last_trace_code_h:org.apache.tez.mapreduce.input.MRInputLegacy:INPUT:0:0], [Reducer 2:org.apache.tez.runtime.library.output.UnorderedPartitionedKVOutput:OUTPUT:1048576:1048576] 2021-01-29 13:32:18,314 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Starting Inputs/Outputs 2021-01-29 13:32:18,314 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Input: dm_common_helper_detect_click_last_trace_code_h being auto started by the framework. Subsequent instances will not be auto-started 2021-01-29 13:32:18,315 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Num IOs determined for AutoStart: 1 2021-01-29 13:32:18,315 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 IOs to start 2021-01-29 13:32:18,315 [INFO] [I/O Setup 1 Start: {dm_common_helper_detect_click_last_trace_code_h}] |runtime.LogicalIOProcessorRuntimeTask|: Started Input with src edge: dm_common_helper_detect_click_last_trace_code_h 2021-01-29 13:32:18,315 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: AutoStartComplete 2021-01-29 13:32:18,316 [INFO] [TezChild] |task.TaskRunner2Callable|: Running task, taskAttemptId=attempt_1611750110647_64528_1_00_000000_0 2021-01-29 13:32:18,383 [INFO] [TezChild] |conf.HiveConf|: Found configuration file null 2021-01-29 13:32:18,502 [INFO] [TezChild] |exec.Utilities|: PLAN PATH = hdfs://ztcluster/tmp/hive/data_quality/fe538bf7-9f34-430a-9619-0b0e1b97b402/hive_2021-01-29_13-31-57_274_5600475007733480004-168252/data_quality/_tez_scratch_dir/2ba4f9da-44e5-4688-aece-e3045e3b3c17/map.xml 2021-01-29 13:32:18,504 [INFO] [TezChild] |exec.SerializationUtilities|: Deserializing MapWork using kryo 2021-01-29 13:32:18,824 [INFO] [TezChild] |exec.Utilities|: Deserialized plan (via RPC) - name: Map 1 size: 4.67KB 2021-01-29 13:32:18,824 [INFO] [TezChild] |tez.ObjectCache|: Caching key: hive_20210129133157_63a0c95d-f2f9-4c13-b252-de43cbc06abc_Map 1__MAP_PLAN__ 2021-01-29 13:32:18,829 [INFO] [TezChild] |tez.MapRecordProcessor|: The input names are: [dm_common_helper_detect_click_last_trace_code_h] 2021-01-29 13:32:18,829 [INFO] [TezChild] |input.MRInput|: Initialized MRInput: dm_common_helper_detect_click_last_trace_code_h 2021-01-29 13:32:18,843 [INFO] [TezChild] |exec.Utilities|: PLAN PATH = hdfs://ztcluster/tmp/hive/data_quality/fe538bf7-9f34-430a-9619-0b0e1b97b402/hive_2021-01-29_13-31-57_274_5600475007733480004-168252/data_quality/_tez_scratch_dir/2ba4f9da-44e5-4688-aece-e3045e3b3c17/map.xml 2021-01-29 13:32:18,857 [INFO] [TezChild] |exec.Utilities|: PLAN PATH = hdfs://ztcluster/tmp/hive/data_quality/fe538bf7-9f34-430a-9619-0b0e1b97b402/hive_2021-01-29_13-31-57_274_5600475007733480004-168252/data_quality/_tez_scratch_dir/2ba4f9da-44e5-4688-aece-e3045e3b3c17/map.xml 2021-01-29 13:32:19,613 [INFO] [TezChild] |impl.OrcCodecPool|: Got brand-new codec ZLIB 2021-01-29 13:32:19,644 [INFO] [TezChild] |exec.Utilities|: PLAN PATH = hdfs://ztcluster/tmp/hive/data_quality/fe538bf7-9f34-430a-9619-0b0e1b97b402/hive_2021-01-29_13-31-57_274_5600475007733480004-168252/data_quality/_tez_scratch_dir/2ba4f9da-44e5-4688-aece-e3045e3b3c17/map.xml 2021-01-29 13:32:19,674 [INFO] [TezChild] |orc.OrcInputFormat|: ORC pushdown predicate: leaf-0 = (EQUALS p_dt 2021-01-29), leaf-1 = (EQUALS p_hours 11), expr = (and leaf-0 leaf-1) 2021-01-29 13:32:19,675 [INFO] [TezChild] |orc.ReaderImpl|: Reading ORC rows from hdfs://ztcluster/user/hive/warehouse/dm_db.db/dm_common_helper_detect_click_last_trace_code_h/p_dt=2021-01-29/p_hours=11/000000_0 with {include: [true, false, false, false], offset: 3, length: 526, sarg: leaf-0 = (EQUALS p_dt 2021-01-29), leaf-1 = (EQUALS p_hours 11), expr = (and leaf-0 leaf-1), columns: ['null', 'null', 'null', 'null'], schema: struct, includeAcidColumns: true} 2021-01-29 13:32:19,714 [INFO] [TezChild] |exec.Utilities|: PLAN PATH = hdfs://ztcluster/tmp/hive/data_quality/fe538bf7-9f34-430a-9619-0b0e1b97b402/hive_2021-01-29_13-31-57_274_5600475007733480004-168252/data_quality/_tez_scratch_dir/2ba4f9da-44e5-4688-aece-e3045e3b3c17/map.xml 2021-01-29 13:32:19,720 [INFO] [TezChild] |lib.MRReaderMapred|: Processing split: TezGroupedSplit{wrappedSplits=[org.apache.hadoop.hive.ql.io.orc.OrcInputFormat:OrcSplit [hdfs://ztcluster/user/hive/warehouse/dm_db.db/dm_common_helper_detect_click_last_trace_code_h/p_dt=2021-01-29/p_hours=11/000000_0, start=3, length=526, isOriginal=true, fileLength=876, hasFooter=false, hasBase=true, deltas=0]], wrappedInputFormatName='org.apache.hadoop.hive.ql.io.HiveInputFormat', locations=[wx12-dsj-node060, wx12-dsj-node053, wx12-dsj-node055], rack='null', length=526} 2021-01-29 13:32:19,725 [INFO] [TezChild] |input.MRInput|: dm_common_helper_detect_click_last_trace_code_h initialized RecordReader from event 2021-01-29 13:32:19,732 [INFO] [TezChild] |writers.BaseUnorderedPartitionedKVWriter|: Instantiating Partitioner: [org.apache.tez.mapreduce.partition.MRPartitioner] 2021-01-29 13:32:19,778 [INFO] [TezChild] |compress.CodecPool|: Got brand-new compressor [.snappy] 2021-01-29 13:32:19,780 [INFO] [TezChild] |writers.UnorderedPartitionedKVWriter|: Reducer_2: numBuffers=2, sizePerBuffer=524288, skipBuffers=true, numPartitions=1, availableMemory=1048576, maxSingleBufferSizeBytes=2147483647, pipelinedShuffle=false, isFinalMergeEnabled=true, numPartitions=1, reportPartitionStats=MEMORY_OPTIMIZED, dataViaEventsEnabled=false, dataViaEventsMaxSize=512, useCachedStreamConfig=true, useCachedStream=false 2021-01-29 13:32:19,782 [INFO] [TezChild] |tez.MapRecordProcessor|: Main input name is Map 1 2021-01-29 13:32:19,782 [INFO] [TezChild] |vector.VectorMapOperator|: Initializing operator MAP[0] 2021-01-29 13:32:19,784 [INFO] [TezChild] |tez.MapRecordProcessor|: Id =0 null <\Children> <\Parent> <\MAP> 2021-01-29 13:32:19,785 [INFO] [TezChild] |tez.MapRecordProcessor|: Memory available for operators set to 5.60GB 2021-01-29 13:32:19,789 [INFO] [TezChild] |tez.ObjectCache|: Caching key: hive_20210129133157_63a0c95d-f2f9-4c13-b252-de43cbc06abc_DynamicValueRegistry 2021-01-29 13:32:19,790 [INFO] [TezChild] |exec.TableScanOperator|: Initializing operator TS[0] 2021-01-29 13:32:19,790 [INFO] [TezChild] |vector.VectorSelectOperator|: Initializing operator SEL[9] 2021-01-29 13:32:19,791 [INFO] [TezChild] |vector.VectorGroupByOperator|: Initializing operator GBY[10] 2021-01-29 13:32:19,799 [INFO] [TezChild] |reducesink.VectorReduceSinkEmptyKeyOperator|: Initializing operator RS[11] 2021-01-29 13:32:19,799 [INFO] [TezChild] |reducesink.VectorReduceSinkCommonOperator|: Using tag = -1 2021-01-29 13:32:19,801 [INFO] [TezChild] |exec.OperatorUtils|: Setting output collector: RS[11] --> Reducer 2 2021-01-29 13:32:19,811 [INFO] [TezChild] |tez.ObjectCache|: Releasing key: hive_20210129133157_63a0c95d-f2f9-4c13-b252-de43cbc06abc_Map 1__MAP_PLAN__ 2021-01-29 13:32:19,811 [INFO] [TezChild] |tez.ObjectCache|: Releasing key: hive_20210129133157_63a0c95d-f2f9-4c13-b252-de43cbc06abc_DynamicValueRegistry 2021-01-29 13:32:19,812 [INFO] [TezChild] |vector.VectorMapOperator|: RECORDS_OUT_INTERMEDIATE_Map_1:0, DESERIALIZE_ERRORS:0, RECORDS_IN_Map_1:10, RECORDS_OUT_OPERATOR_MAP_0:0, 2021-01-29 13:32:19,812 [INFO] [TezChild] |exec.TableScanOperator|: RECORDS_OUT_INTERMEDIATE_Map_1:0, RECORDS_OUT_OPERATOR_TS_0:10, 2021-01-29 13:32:19,812 [INFO] [TezChild] |vector.VectorSelectOperator|: RECORDS_OUT_INTERMEDIATE_Map_1:0, RECORDS_OUT_OPERATOR_SEL_9:10, 2021-01-29 13:32:19,816 [INFO] [TezChild] |reducesink.VectorReduceSinkCommonOperator|: RS[11]: records written - 1 2021-01-29 13:32:19,817 [INFO] [TezChild] |vector.VectorGroupByOperator|: RECORDS_OUT_INTERMEDIATE_Map_1:0, RECORDS_OUT_OPERATOR_GBY_10:1, 2021-01-29 13:32:19,817 [INFO] [TezChild] |reducesink.VectorReduceSinkCommonOperator|: RS[11]: records written - 1 2021-01-29 13:32:19,817 [INFO] [TezChild] |reducesink.VectorReduceSinkEmptyKeyOperator|: RECORDS_OUT_INTERMEDIATE_Map_1:1, RECORDS_OUT_OPERATOR_RS_11:1, 2021-01-29 13:32:19,821 [INFO] [TezChild] |task.TaskRunner2Callable|: Closing task, taskAttemptId=attempt_1611750110647_64528_1_00_000000_0 2021-01-29 13:32:19,822 [INFO] [TezChild] |writers.UnorderedPartitionedKVWriter|: Reducer_2: Waiting for all spills to complete : Pending : 0 2021-01-29 13:32:19,822 [INFO] [TezChild] |writers.UnorderedPartitionedKVWriter|: Reducer_2: All spills complete 2021-01-29 13:32:19,850 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Joining on EventRouter 2021-01-29 13:32:19,853 [INFO] [TezChild] |task.TaskRunner2Callable|: Task completed, taskAttemptId=attempt_1611750110647_64528_1_00_000000_0, askedToStop=false 2021-01-29 13:32:19,854 [INFO] [TezChild] |task.TaskRunner2Callable|: Cleaning up task attempt_1611750110647_64528_1_00_000000_0, stopRequested=false 2021-01-29 13:32:19,856 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1611750110647_64528_1_00_000000_0: Counters: 41 [[File System Counters FILE_BYTES_WRITTEN=57, HDFS_BYTES_READ=1034, HDFS_READ_OPS=1, HDFS_OP_OPEN=1][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=0, GC_TIME_MILLIS=66, CPU_MILLISECONDS=6270, PHYSICAL_MEMORY_BYTES=2147483648, VIRTUAL_MEMORY_BYTES=8360558592, COMMITTED_HEAP_BYTES=2147483648, INPUT_RECORDS_PROCESSED=2, INPUT_SPLIT_LENGTH_BYTES=526, OUTPUT_RECORDS=1, OUTPUT_LARGE_RECORDS=0, OUTPUT_BYTES=3, OUTPUT_BYTES_WITH_OVERHEAD=11, OUTPUT_BYTES_PHYSICAL=49, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, DATA_BYTES_VIA_EVENT=0][HIVE DESERIALIZE_ERRORS=0, RECORDS_IN_Map_1=10, RECORDS_OUT_INTERMEDIATE_Map_1=1, RECORDS_OUT_OPERATOR_GBY_10=1, RECORDS_OUT_OPERATOR_MAP_0=0, RECORDS_OUT_OPERATOR_RS_11=1, RECORDS_OUT_OPERATOR_SEL_9=10, RECORDS_OUT_OPERATOR_TS_0=10][TaskCounter_Map_1_INPUT_dm_common_helper_detect_click_last_trace INPUT_RECORDS_PROCESSED=2, INPUT_SPLIT_LENGTH_BYTES=526][TaskCounter_Map_1_OUTPUT_Reducer_2 ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILL_COUNT=0, DATA_BYTES_VIA_EVENT=0, OUTPUT_BYTES=3, OUTPUT_BYTES_PHYSICAL=49, OUTPUT_BYTES_WITH_OVERHEAD=11, OUTPUT_LARGE_RECORDS=0, OUTPUT_RECORDS=1, SPILLED_RECORDS=0]] 2021-01-29 13:32:19,891 [INFO] [main] |task.TezTaskRunner2|: TaskRunnerResult for attempt_1611750110647_64528_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, containerShutdownRequested=false} 2021-01-29 13:32:19,891 [INFO] [main] |task.TezChild|: TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, containerShutdownRequested=false} 2021-01-29 13:32:19,902 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: End of LogType:syslog_attempt_1611750110647_64528_1_00_000000_0 ***************************************************************************************************************** Container: container_e703_1611750110647_64528_01_000002 on wx12-dsj-node008_45454_1611898349442 LogAggregationType: AGGREGATED =============================================================================================== LogType:syslog_attempt_1611750110647_64528_1_01_000000_0 LogLastModifiedTime:Fri Jan 29 13:32:29 +0800 2021 LogLength:13292 LogContents: 2021-01-29 13:32:19,998 [INFO] [main] |metrics.TaskCounterUpdater|: Using ResourceCalculatorProcessTree : org.apache.tez.util.TezMxBeanResourceCalculator 2021-01-29 13:32:19,998 [INFO] [main] |runtime.LogicalIOProcessorRuntimeTask|: Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : select count(1) as counts, 0 a...'11' (Stage-1), VertexName: Reducer 2, VertexParallelism: 1, TaskAttemptID:attempt_1611750110647_64528_1_01_000000_0, processorName=org.apache.hadoop.hive.ql.exec.tez.ReduceTezProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=Map 1, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.UnorderedKVInput }}, ], outputSpecList=[{{ destinationVertexName=out_Reducer 2, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ], taskConfEntryCount=0 2021-01-29 13:32:19,998 [INFO] [main] |resources.MemoryDistributor|: InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=6012534784, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor 2021-01-29 13:32:19,999 [INFO] [TezChild] |task.TaskRunner2Callable|: Initializing task, taskAttemptId=attempt_1611750110647_64528_1_01_000000_0 2021-01-29 13:32:20,008 [INFO] [TaskHeartbeatThread] |task.TaskReporter|: Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1611750110647_64528_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1 2021-01-29 13:32:20,021 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor 2021-01-29 13:32:20,021 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish 2021-01-29 13:32:20,021 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish 2021-01-29 13:32:20,026 [INFO] [I/O Setup 1 Initialize: {out_Reducer 2}] |output.MROutput|: out_Reducer 2: outputFormat=org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl, using newmapreduce API=false 2021-01-29 13:32:20,030 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: All initializers finished 2021-01-29 13:32:20,030 [INFO] [TezChild] |resources.MemoryDistributor|: InitialRequests=[out_Reducer 2:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [Map 1:INPUT:1288490240:org.apache.tez.runtime.library.input.UnorderedKVInput] 2021-01-29 13:32:20,030 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1] 2021-01-29 13:32:20,030 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996 2021-01-29 13:32:20,031 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: Scaling Requests. NumRequests: 2, numScaledRequests: 2, TotalRequested: 1288490240, TotalRequestedScaled: 6.4424512E8, TotalJVMHeap: 6012534784, TotalAvailable: 4028398305, TotalRequested/TotalJVMHeap:0.21 2021-01-29 13:32:20,031 [INFO] [TezChild] |resources.MemoryDistributor|: Allocations=[out_Reducer 2:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [Map 1:org.apache.tez.runtime.library.input.UnorderedKVInput:INPUT:1288490240:1288490240] 2021-01-29 13:32:20,031 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Starting Inputs/Outputs 2021-01-29 13:32:20,031 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Input: Map 1 being auto started by the framework. Subsequent instances will not be auto-started 2021-01-29 13:32:20,031 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Num IOs determined for AutoStart: 1 2021-01-29 13:32:20,031 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 IOs to start 2021-01-29 13:32:20,031 [INFO] [I/O Setup 0 Start: {Map 1}] |impl.SimpleFetchedInputAllocator|: Map_1: RequestedMemory=1288490240, AssignedMemory=1288490240, maxSingleShuffleLimit=322122560 2021-01-29 13:32:20,136 [INFO] [I/O Setup 0 Start: {Map 1}] |impl.ShuffleManager|: Map_1: numInputs=1, compressionCodec=org.apache.hadoop.io.compress.SnappyCodec, numFetchers=1, ifileBufferSize=131072, ifileReadAheadEnabled=true, ifileReadAheadLength=4194304, localDiskFetchEnabled=true, sharedFetchEnabled=false, keepAlive=false, keepAliveMaxConnections=20, connectionTimeout=180000, readTimeout=180000, bufferSize=8192, bufferSize=8192, maxTaskOutputAtOnce=20 2021-01-29 13:32:20,141 [INFO] [I/O Setup 0 Start: {Map 1}] |runtime.LogicalIOProcessorRuntimeTask|: Started Input with src edge: Map 1 2021-01-29 13:32:20,141 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: AutoStartComplete 2021-01-29 13:32:20,142 [INFO] [TezChild] |task.TaskRunner2Callable|: Running task, taskAttemptId=attempt_1611750110647_64528_1_01_000000_0 2021-01-29 13:32:20,144 [INFO] [TezChild] |exec.Utilities|: PLAN PATH = hdfs://ztcluster/tmp/hive/data_quality/fe538bf7-9f34-430a-9619-0b0e1b97b402/hive_2021-01-29_13-31-57_274_5600475007733480004-168252/data_quality/_tez_scratch_dir/46384c57-e7c4-49f4-9a3a-541a1f608002/reduce.xml 2021-01-29 13:32:20,145 [INFO] [TezTaskEventRouter{attempt_1611750110647_64528_1_01_000000_0}] |impl.ShuffleInputEventHandlerImpl|: Map 1: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0 2021-01-29 13:32:20,146 [INFO] [TezChild] |exec.SerializationUtilities|: Deserializing ReduceWork using kryo 2021-01-29 13:32:20,160 [INFO] [TezChild] |exec.Utilities|: Deserialized plan (via RPC) - name: Reducer 2 size: 2.21KB 2021-01-29 13:32:20,160 [INFO] [TezChild] |tez.ObjectCache|: Caching key: hive_20210129133157_63a0c95d-f2f9-4c13-b252-de43cbc06abc_Reducer 2__REDUCE_PLAN__ 2021-01-29 13:32:20,163 [INFO] [TezChild] |tez.ReduceRecordProcessor|: Waiting for ShuffleInputs to become ready 2021-01-29 13:32:20,165 [INFO] [Fetcher_B {Map_1} #0] |ShuffleManager.fetch|: Completed fetch for attempt: {0, 0, attempt_1611750110647_64528_1_00_000000_0_10002} to DISK_DIRECT, csize=25, dsize=11, EndTime=1611898340165, TimeTaken=7, Rate=0.00 MB/s 2021-01-29 13:32:20,166 [INFO] [Fetcher_B {Map_1} #0] |impl.ShuffleManager|: All inputs fetched for input vertex : Map 1 2021-01-29 13:32:20,166 [INFO] [TezChild] |tez.ReduceRecordProcessor|: Main work is Reducer 2 2021-01-29 13:32:20,166 [INFO] [Fetcher_B {Map_1} #0] |impl.ShuffleManager|: copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s) 2021-01-29 13:32:20,166 [INFO] [TezChild] |tez.ReduceRecordProcessor|: Memory available for operators set to 5.60GB 2021-01-29 13:32:20,166 [INFO] [ShuffleRunner {Map_1}] |impl.ShuffleManager|: Map_1: Shutting down FetchScheduler, Was Interrupted: false 2021-01-29 13:32:20,166 [INFO] [TezChild] |tez.ObjectCache|: Caching key: hive_20210129133157_63a0c95d-f2f9-4c13-b252-de43cbc06abc_DynamicValueRegistry 2021-01-29 13:32:20,166 [INFO] [ShuffleRunner {Map_1}] |impl.ShuffleManager|: Map_1: Scheduler thread completed 2021-01-29 13:32:20,188 [INFO] [TezChild] |vector.VectorGroupByOperator|: Initializing operator GBY[12] 2021-01-29 13:32:20,189 [INFO] [TezChild] |vector.VectorGroupByOperator|: using global aggregation processing mode 2021-01-29 13:32:20,189 [INFO] [TezChild] |vector.VectorSelectOperator|: Initializing operator SEL[13] 2021-01-29 13:32:20,198 [INFO] [TezChild] |vector.VectorFileSinkOperator|: Initializing operator FS[14] 2021-01-29 13:32:20,353 [INFO] [TezChild] |exec.FileSinkOperator|: Using serializer : class org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe[[[B@59593f63]:[_col0, _col1, _col2]:[bigint, int, int]] and formatter : org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat@14859b4 2021-01-29 13:32:20,353 [INFO] [TezChild] |Configuration.deprecation|: mapred.healthChecker.script.timeout is deprecated. Instead, use mapreduce.tasktracker.healthchecker.script.timeout 2021-01-29 13:32:20,356 [INFO] [TezChild] |tez.ReduceRecordProcessor|: Id =12 Id =13 Id =14 <\Children> Id = 13 null<\Parent> <\FS> <\Children> Id = 12 null<\Parent> <\SEL> <\Children> <\Parent> <\GBY> 2021-01-29 13:32:20,356 [INFO] [TezChild] |tez.ReduceRecordProcessor|: Starting Output: out_Reducer 2 2021-01-29 13:32:20,362 [INFO] [TezChild] |compress.CodecPool|: Got brand-new decompressor [.snappy] 2021-01-29 13:32:20,364 [INFO] [TezChild] |readers.UnorderedKVReader|: Num Records read: 1 2021-01-29 13:32:20,364 [INFO] [TezChild] |tez.ObjectCache|: Releasing key: hive_20210129133157_63a0c95d-f2f9-4c13-b252-de43cbc06abc_Reducer 2__REDUCE_PLAN__ 2021-01-29 13:32:20,364 [INFO] [TezChild] |tez.ObjectCache|: Releasing key: hive_20210129133157_63a0c95d-f2f9-4c13-b252-de43cbc06abc_DynamicValueRegistry 2021-01-29 13:32:20,365 [INFO] [TezChild] |exec.FileSinkOperator|: Final Path: FS hdfs://ztcluster/tmp/hive/data_quality/fe538bf7-9f34-430a-9619-0b0e1b97b402/hive_2021-01-29_13-31-57_274_5600475007733480004-168248/-mr-10001/.hive-staging_hive_2021-01-29_13-31-57_274_5600475007733480004-168248/_tmp.-ext-10002/000000_0 2021-01-29 13:32:20,366 [INFO] [TezChild] |exec.FileSinkOperator|: Writing to temp file: FS hdfs://ztcluster/tmp/hive/data_quality/fe538bf7-9f34-430a-9619-0b0e1b97b402/hive_2021-01-29_13-31-57_274_5600475007733480004-168248/-mr-10001/.hive-staging_hive_2021-01-29_13-31-57_274_5600475007733480004-168248/_task_tmp.-ext-10002/_tmp.000000_0 2021-01-29 13:32:20,366 [INFO] [TezChild] |exec.FileSinkOperator|: New Final Path: FS hdfs://ztcluster/tmp/hive/data_quality/fe538bf7-9f34-430a-9619-0b0e1b97b402/hive_2021-01-29_13-31-57_274_5600475007733480004-168248/-mr-10001/.hive-staging_hive_2021-01-29_13-31-57_274_5600475007733480004-168248/_tmp.-ext-10002/000000_0 2021-01-29 13:32:20,428 [INFO] [TezChild] |exec.FileSinkOperator|: FS[14]: records written - 1 2021-01-29 13:32:20,428 [INFO] [TezChild] |vector.VectorGroupByOperator|: RECORDS_OUT_OPERATOR_GBY_12:1, RECORDS_OUT_INTERMEDIATE_Reducer_2:0, 2021-01-29 13:32:20,429 [INFO] [TezChild] |vector.VectorSelectOperator|: RECORDS_OUT_OPERATOR_SEL_13:1, RECORDS_OUT_INTERMEDIATE_Reducer_2:0, 2021-01-29 13:32:20,429 [INFO] [TezChild] |exec.FileSinkOperator|: FS[14]: records written - 1 2021-01-29 13:32:20,524 [INFO] [TezChild] |vector.VectorFileSinkOperator|: RECORDS_OUT_OPERATOR_FS_14:1, RECORDS_OUT_INTERMEDIATE_Reducer_2:0, RECORDS_OUT_0:1, 2021-01-29 13:32:20,525 [INFO] [TezChild] |task.TaskRunner2Callable|: Closing task, taskAttemptId=attempt_1611750110647_64528_1_01_000000_0 2021-01-29 13:32:20,525 [INFO] [TezChild] |impl.ShuffleInputEventHandlerImpl|: Map 1: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose 2021-01-29 13:32:20,525 [INFO] [TezChild] |impl.ShuffleManager|: Shutting down pending fetchers on sourceMap_1: 0 2021-01-29 13:32:20,525 [INFO] [TezChild] |output.MROutput|: out_Reducer 2 closed 2021-01-29 13:32:20,525 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Joining on EventRouter 2021-01-29 13:32:20,527 [INFO] [TezChild] |task.TaskRunner2Callable|: Task completed, taskAttemptId=attempt_1611750110647_64528_1_01_000000_0, askedToStop=false 2021-01-29 13:32:20,527 [INFO] [TezChild] |task.TaskRunner2Callable|: Cleaning up task attempt_1611750110647_64528_1_01_000000_0, stopRequested=false 2021-01-29 13:32:20,528 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1611750110647_64528_1_01_000000_0: Counters: 42 [[File System Counters FILE_BYTES_READ=57, HDFS_BYTES_WRITTEN=106, HDFS_READ_OPS=3, HDFS_WRITE_OPS=2, HDFS_OP_CREATE=1, HDFS_OP_GET_FILE_STATUS=3, HDFS_OP_RENAME=1][org.apache.tez.common.counters.TaskCounter NUM_SHUFFLED_INPUTS=1, NUM_FAILED_SHUFFLE_INPUTS=0, GC_TIME_MILLIS=66, CPU_MILLISECONDS=2630, PHYSICAL_MEMORY_BYTES=2147483648, VIRTUAL_MEMORY_BYTES=8365252608, COMMITTED_HEAP_BYTES=2147483648, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=0, SHUFFLE_BYTES=25, SHUFFLE_BYTES_DECOMPRESSED=11, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=25, SHUFFLE_PHASE_TIME=123, FIRST_EVENT_RECEIVED=102, LAST_EVENT_RECEIVED=102][HIVE CREATED_FILES=1, RECORDS_OUT_0=1, RECORDS_OUT_INTERMEDIATE_Reducer_2=0, RECORDS_OUT_OPERATOR_FS_14=1, RECORDS_OUT_OPERATOR_GBY_12=1, RECORDS_OUT_OPERATOR_SEL_13=1][TaskCounter_Reducer_2_INPUT_Map_1 FIRST_EVENT_RECEIVED=102, INPUT_RECORDS_PROCESSED=1, LAST_EVENT_RECEIVED=102, NUM_FAILED_SHUFFLE_INPUTS=0, NUM_SHUFFLED_INPUTS=1, SHUFFLE_BYTES=25, SHUFFLE_BYTES_DECOMPRESSED=11, SHUFFLE_BYTES_DISK_DIRECT=25, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_PHASE_TIME=123][TaskCounter_Reducer_2_OUTPUT_out_Reducer_2 OUTPUT_RECORDS=0]] 2021-01-29 13:32:20,536 [INFO] [main] |task.TezTaskRunner2|: TaskRunnerResult for attempt_1611750110647_64528_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, containerShutdownRequested=false} 2021-01-29 13:32:20,537 [INFO] [main] |task.TezChild|: TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, containerShutdownRequested=false} 2021-01-29 13:32:20,549 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: End of LogType:syslog_attempt_1611750110647_64528_1_01_000000_0 ***************************************************************************************************************** Container: container_e703_1611750110647_64528_01_000001 on wx12-dsj-node045_45454_1611898349701 LogAggregationType: AGGREGATED =============================================================================================== LogType:container-localizer-syslog LogLastModifiedTime:Fri Jan 29 13:32:30 +0800 2021 LogLength:355 LogContents: 2021-01-29 13:32:03,706 WARN [main] org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer: Localization running as yarn not data_quality 2021-01-29 13:32:03,801 INFO [main] org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer: Disk Validator: yarn.nodemanager.disk-validator is loaded. End of LogType:container-localizer-syslog ******************************************************************************************* Container: container_e703_1611750110647_64528_01_000001 on wx12-dsj-node045_45454_1611898349701 LogAggregationType: AGGREGATED =============================================================================================== LogType:dag_1611750110647_64528_1.dot LogLastModifiedTime:Fri Jan 29 13:32:30 +0800 2021 LogLength:1448 LogContents: digraph select_count_1_as_counts_0_a_11_Stage_1_ { graph [ label="select_count_1_as_counts_0_a_11_Stage_1_", fontsize=24, fontname=Helvetica]; node [fontsize=12, fontname=Helvetica]; edge [fontsize=9, fontcolor=blue, fontname=Arial]; "select_count_1_as_counts_0_a_11_Stage_1_.Map_1_dm_common_helper_detect_click_last_trace_code_h" [ label = "Map_1[dm_common_helper_detect_click_last_trace_code_h]", shape = "box" , color= "black"]; "select_count_1_as_counts_0_a_11_Stage_1_.Map_1_dm_common_helper_detect_click_last_trace_code_h" -> "select_count_1_as_counts_0_a_11_Stage_1_.Map_1" [ label = "Input [inputClass=MRInputLegacy,\n initializer=HiveSplitGenerator]" ]; "select_count_1_as_counts_0_a_11_Stage_1_.Map_1" [ label = "Map_1[MapTezProcessor]" , color= "black" ]; "select_count_1_as_counts_0_a_11_Stage_1_.Map_1" -> "select_count_1_as_counts_0_a_11_Stage_1_.Reducer_2" [ label = "[input=UnorderedPartitionedKVOutput,\n output=UnorderedKVInput,\n dataMovement=SCATTER_GATHER,\n schedulingType=SEQUENTIAL]" ]; "select_count_1_as_counts_0_a_11_Stage_1_.Reducer_2" [ label = "Reducer_2[ReduceTezProcessor]" , color= "black" ]; "select_count_1_as_counts_0_a_11_Stage_1_.Reducer_2" -> "select_count_1_as_counts_0_a_11_Stage_1_.Reducer_2_out_Reducer_2" [ label = "Output [outputClass=MROutput,\n committer=]" ]; "select_count_1_as_counts_0_a_11_Stage_1_.Reducer_2_out_Reducer_2" [ label = "Reducer_2[out_Reducer_2]", shape = "box" , color= "black"]; } End of LogType:dag_1611750110647_64528_1.dot ********************************************************************************************** Container: container_e703_1611750110647_64528_01_000001 on wx12-dsj-node045_45454_1611898349701 LogAggregationType: AGGREGATED =============================================================================================== LogType:directory.info LogLastModifiedTime:Fri Jan 29 13:32:30 +0800 2021 LogLength:23424 LogContents: ls -l: total 12 -rw------- 1 yarn hadoop 178 Jan 29 13:32 container_tokens lrwxrwxrwx 1 yarn hadoop 158 Jan 29 13:32 hive-exec-3.1.0.3.1.5.0-152-52ac635d1c2eaf5d3e864f9c6dd1cc6c640a4133a03a6747e20ad2156829ebdf.jar -> /data01/hdp/yarn/local/usercache/data_quality/filecache/19219/hive-exec-3.1.0.3.1.5.0-152-52ac635d1c2eaf5d3e864f9c6dd1cc6c640a4133a03a6747e20ad2156829ebdf.jar lrwxrwxrwx 1 yarn hadoop 84 Jan 29 13:32 hive-hcatalog-core.jar -> /data01/hdp/yarn/local/usercache/data_quality/filecache/45853/hive-hcatalog-core.jar -rwx------ 1 yarn hadoop 5702 Jan 29 13:32 launch_container.sh lrwxrwxrwx 1 yarn hadoop 111 Jan 29 13:32 tez-conf.pb -> /data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/filecache/11/tez-conf.pb lrwxrwxrwx 1 yarn hadoop 48 Jan 29 13:32 tezlib -> /data01/hdp/yarn/local/filecache/2160/tez.tar.gz lrwxrwxrwx 1 yarn hadoop 130 Jan 29 13:32 tez.session.local-resources.pb -> /data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/filecache/10/tez.session.local-resources.pb drwxr-s--- 2 yarn hadoop 6 Jan 29 13:32 tmp find -L . -maxdepth 5 -ls: 5019326 0 drwxr-s--- 3 yarn hadoop 309 Jan 29 13:32 . 2150202054 0 drwxr-s--- 2 yarn hadoop 6 Jan 29 13:32 ./tmp 5019327 8 -rwx------ 1 yarn hadoop 5702 Jan 29 13:32 ./launch_container.sh 5019376 4 -rw------- 1 yarn hadoop 178 Jan 29 13:32 ./container_tokens 6444171974 4 -r-x------ 1 yarn hadoop 688 Jan 29 13:32 ./tez.session.local-resources.pb 2147492889 4 drwxr-xr-x 3 yarn hadoop 4096 May 28 2020 ./tezlib 2147492890 4 -r-xr-xr-x 1 yarn hadoop 1509 Dec 13 2019 ./tezlib/NOTICE 2147492891 4 -r-xr-xr-x 1 yarn hadoop 1045 Dec 13 2019 ./tezlib/LICENSE-MIT 2147492892 36 -r-xr-xr-x 1 yarn hadoop 36261 Dec 13 2019 ./tezlib/LICENSE-CDDLv1.1-GPLv2_withCPE 2147492893 20 -r-xr-xr-x 1 yarn hadoop 16792 Dec 13 2019 ./tezlib/LICENSE-CDDLv1.0 2147492894 4 -r-xr-xr-x 1 yarn hadoop 1465 Dec 13 2019 ./tezlib/LICENSE-BSD-3clause 2147492895 20 -r-xr-xr-x 1 yarn hadoop 17257 Dec 13 2019 ./tezlib/LICENSE 2147492827 8 -r-xr-xr-x 1 yarn hadoop 4128 Dec 13 2019 ./tezlib/LICENSE-SIL_OpenFontLicense-v1.1 2147492928 12 -r-xr-xr-x 1 yarn hadoop 8897 Dec 13 2019 ./tezlib/hadoop-shim-0.9.1.3.1.5.0-152.jar 2147492929 1020 -r-xr-xr-x 1 yarn hadoop 1042290 Dec 13 2019 ./tezlib/tez-api-0.9.1.3.1.5.0-152.jar 2147492931 84 -r-xr-xr-x 1 yarn hadoop 85945 Dec 13 2019 ./tezlib/tez-common-0.9.1.3.1.5.0-152.jar 2147492932 196 -r-xr-xr-x 1 yarn hadoop 199575 Dec 13 2019 ./tezlib/tez-runtime-internals-0.9.1.3.1.5.0-152.jar 2147492933 764 -r-xr-xr-x 1 yarn hadoop 780915 Dec 13 2019 ./tezlib/tez-runtime-library-0.9.1.3.1.5.0-152.jar 2147492934 292 -r-xr-xr-x 1 yarn hadoop 296388 Dec 13 2019 ./tezlib/tez-mapreduce-0.9.1.3.1.5.0-152.jar 2147492935 56 -r-xr-xr-x 1 yarn hadoop 56879 Dec 13 2019 ./tezlib/tez-examples-0.9.1.3.1.5.0-152.jar 2147492936 1380 -r-xr-xr-x 1 yarn hadoop 1410092 Dec 13 2019 ./tezlib/tez-dag-0.9.1.3.1.5.0-152.jar 2147492938 156 -r-xr-xr-x 1 yarn hadoop 158853 Dec 13 2019 ./tezlib/tez-tests-0.9.1.3.1.5.0-152.jar 2147492939 108 -r-xr-xr-x 1 yarn hadoop 107600 Dec 13 2019 ./tezlib/tez-ext-service-tests-0.9.1.3.1.5.0-152.jar 2147492941 121292 -r-xr-xr-x 1 yarn hadoop 124202048 Dec 13 2019 ./tezlib/tez-ui-0.9.1.3.1.5.0-152.war 2147492976 76 -r-xr-xr-x 1 yarn hadoop 75759 Dec 13 2019 ./tezlib/tez-protobuf-history-plugin-0.9.1.3.1.5.0-152.jar 2147492999 28 -r-xr-xr-x 1 yarn hadoop 28274 Dec 13 2019 ./tezlib/tez-yarn-timeline-history-0.9.1.3.1.5.0-152.jar 2147493000 8 -r-xr-xr-x 1 yarn hadoop 7839 Dec 13 2019 ./tezlib/tez-yarn-timeline-history-with-acls-0.9.1.3.1.5.0-152.jar 2147493001 12 -r-xr-xr-x 1 yarn hadoop 11564 Dec 13 2019 ./tezlib/tez-yarn-timeline-cache-plugin-0.9.1.3.1.5.0-152.jar 2147493002 24 -r-xr-xr-x 1 yarn hadoop 23692 Dec 13 2019 ./tezlib/tez-yarn-timeline-history-with-fs-0.9.1.3.1.5.0-152.jar 2147493003 80 -r-xr-xr-x 1 yarn hadoop 79219 Dec 13 2019 ./tezlib/tez-history-parser-0.9.1.3.1.5.0-152.jar 2147493004 80 -r-xr-xr-x 1 yarn hadoop 79108 Dec 13 2019 ./tezlib/tez-job-analyzer-0.9.1.3.1.5.0-152.jar 2147493005 16 -r-xr-xr-x 1 yarn hadoop 15352 Dec 13 2019 ./tezlib/tez-javadoc-tools-0.9.1.3.1.5.0-152.jar 2147493006 8 -r-xr-xr-x 1 yarn hadoop 6443 Dec 13 2019 ./tezlib/hadoop-shim-2.8-0.9.1.3.1.5.0-152.jar 6442920261 12 drwxr-xr-x 3 yarn hadoop 8192 May 28 2020 ./tezlib/lib 6442920262 32 -r-xr-xr-x 1 yarn hadoop 32119 Oct 22 2016 ./tezlib/lib/slf4j-api-1.7.10.jar 6442920263 3028 -r-xr-xr-x 1 yarn hadoop 3099479 Dec 13 2019 ./tezlib/lib/hadoop-yarn-api-3.1.1.3.1.5.0-152.jar 6442920264 280 -r-xr-xr-x 1 yarn hadoop 284220 Oct 22 2016 ./tezlib/lib/commons-lang-2.6.jar 6442920265 2684 -r-xr-xr-x 1 yarn hadoop 2746681 Dec 13 2019 ./tezlib/lib/guava-28.0-jre.jar 6442920266 8 -r-xr-xr-x 1 yarn hadoop 4617 Dec 13 2019 ./tezlib/lib/failureaccess-1.0.1.jar 6442920267 4 -r-xr-xr-x 1 yarn hadoop 2199 Dec 13 2019 ./tezlib/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar 6442920268 36 -r-xr-xr-x 1 yarn hadoop 33031 Dec 13 2019 ./tezlib/lib/jsr305-3.0.0.jar 6442920269 196 -r-xr-xr-x 1 yarn hadoop 200629 Dec 5 2019 ./tezlib/lib/checker-qual-2.8.1.jar 6442920270 16 -r-xr-xr-x 1 yarn hadoop 13166 Dec 13 2019 ./tezlib/lib/error_prone_annotations-2.3.2.jar 6442920271 12 -r-xr-xr-x 1 yarn hadoop 8781 Dec 13 2019 ./tezlib/lib/j2objc-annotations-1.3.jar 6442920272 4 -r-xr-xr-x 1 yarn hadoop 3448 Dec 5 2019 ./tezlib/lib/animal-sniffer-annotations-1.17.jar 6442920273 64 -r-xr-xr-x 1 yarn hadoop 62050 Oct 22 2016 ./tezlib/lib/commons-logging-1.1.3.jar 6442920274 100 -r-xr-xr-x 1 yarn hadoop 102244 Dec 5 2019 ./tezlib/lib/jaxb-api-2.2.11.jar 6442920275 524 -r-xr-xr-x 1 yarn hadoop 533455 Dec 13 2019 ./tezlib/lib/protobuf-java-2.5.0.jar 6442920276 68 -r-xr-xr-x 1 yarn hadoop 67889 Dec 6 2019 ./tezlib/lib/jackson-annotations-2.10.0.jar 6442920277 3956 -r-xr-xr-x 1 yarn hadoop 4050116 Dec 13 2019 ./tezlib/lib/hadoop-common-3.1.1.3.1.5.0-152.jar 6442920278 60 -r-xr-xr-x 1 yarn hadoop 59895 Dec 13 2019 ./tezlib/lib/hadoop-annotations-3.1.1.3.1.5.0-152.jar 6442920279 44 -r-xr-xr-x 1 yarn hadoop 41123 Oct 22 2016 ./tezlib/lib/commons-cli-1.2.jar 6442920280 1564 -r-xr-xr-x 1 yarn hadoop 1599627 Oct 22 2016 ./tezlib/lib/commons-math3-3.1.1.jar 6442920281 720 -r-xr-xr-x 1 yarn hadoop 736658 Oct 22 2016 ./tezlib/lib/httpclient-4.5.2.jar 6442920282 320 -r-xr-xr-x 1 yarn hadoop 326724 Oct 22 2016 ./tezlib/lib/httpcore-4.4.4.jar 6442920283 60 -r-xr-xr-x 1 yarn hadoop 58160 Oct 22 2016 ./tezlib/lib/commons-codec-1.4.jar 6442920284 184 -r-xr-xr-x 1 yarn hadoop 185140 Oct 22 2016 ./tezlib/lib/commons-io-2.4.jar 6442920285 304 -r-xr-xr-x 1 yarn hadoop 307410 Dec 5 2019 ./tezlib/lib/commons-net-3.6.jar 6442920286 576 -r-xr-xr-x 1 yarn hadoop 588337 Oct 22 2016 ./tezlib/lib/commons-collections-3.2.2.jar 6442920287 96 -r-xr-xr-x 1 yarn hadoop 95806 Oct 22 2016 ./tezlib/lib/javax.servlet-api-3.1.0.jar 6442920290 120 -r-xr-xr-x 1 yarn hadoop 118802 Dec 5 2019 ./tezlib/lib/jetty-servlet-9.3.24.v20180605.jar 6442920291 96 -r-xr-xr-x 1 yarn hadoop 95305 Dec 5 2019 ./tezlib/lib/jetty-security-9.3.24.v20180605.jar 6442920292 116 -r-xr-xr-x 1 yarn hadoop 114983 Dec 5 2019 ./tezlib/lib/jetty-webapp-9.3.24.v20180605.jar 6442920293 52 -r-xr-xr-x 1 yarn hadoop 51734 Dec 5 2019 ./tezlib/lib/jetty-xml-9.3.24.v20180605.jar 6442920294 128 -r-xr-xr-x 1 yarn hadoop 128719 Oct 22 2016 ./tezlib/lib/jersey-servlet-1.19.jar 6442920295 164 -r-xr-xr-x 1 yarn hadoop 165345 Oct 22 2016 ./tezlib/lib/jersey-json-1.19.jar 6442920296 80 -r-xr-xr-x 1 yarn hadoop 81743 Oct 22 2016 ./tezlib/lib/jettison-1.3.4.jar 6442920297 872 -r-xr-xr-x 1 yarn hadoop 890168 Oct 22 2016 ./tezlib/lib/jaxb-impl-2.2.3-1.jar 6443006788 224 -r-xr-xr-x 1 yarn hadoop 228286 Oct 22 2016 ./tezlib/lib/jackson-core-asl-1.9.2.jar 6443006789 748 -r-xr-xr-x 1 yarn hadoop 765648 Oct 22 2016 ./tezlib/lib/jackson-mapper-asl-1.9.2.jar 6443006792 20 -r-xr-xr-x 1 yarn hadoop 18323 Oct 22 2016 ./tezlib/lib/jackson-jaxrs-1.9.2.jar 6443006793 28 -r-xr-xr-x 1 yarn hadoop 27075 Oct 22 2016 ./tezlib/lib/jackson-xc-1.9.2.jar 6443006794 480 -r-xr-xr-x 1 yarn hadoop 489884 Oct 22 2016 ./tezlib/lib/log4j-1.2.17.jar 6443006795 244 -r-xr-xr-x 1 yarn hadoop 246918 Dec 6 2019 ./tezlib/lib/commons-beanutils-1.9.4.jar 6443006812 604 -r-xr-xr-x 1 yarn hadoop 616888 Dec 5 2019 ./tezlib/lib/commons-configuration2-2.1.1.jar 6443006813 428 -r-xr-xr-x 1 yarn hadoop 434678 Oct 22 2016 ./tezlib/lib/commons-lang3-3.4.jar 6443006814 12 -r-xr-xr-x 1 yarn hadoop 8866 Oct 22 2016 ./tezlib/lib/slf4j-log4j12-1.7.10.jar 6443006815 428 -r-xr-xr-x 1 yarn hadoop 436303 Oct 22 2016 ./tezlib/lib/avro-1.7.7.jar 6443006816 32 -r-xr-xr-x 1 yarn hadoop 29555 Oct 22 2016 ./tezlib/lib/paranamer-2.3.jar 6443006817 1224 -r-xr-xr-x 1 yarn hadoop 1251514 Oct 22 2016 ./tezlib/lib/snappy-java-1.0.5.jar 6443006832 236 -r-xr-xr-x 1 yarn hadoop 241367 Oct 22 2016 ./tezlib/lib/commons-compress-1.4.1.jar 6443006833 96 -r-xr-xr-x 1 yarn hadoop 94672 Oct 22 2016 ./tezlib/lib/xz-1.0.jar 6443006844 128 -r-xr-xr-x 1 yarn hadoop 128414 Dec 13 2019 ./tezlib/lib/re2j-1.1.jar 6443006845 188 -r-xr-xr-x 1 yarn hadoop 190432 Dec 13 2019 ./tezlib/lib/gson-2.2.4.jar 6443032002 136 -r-xr-xr-x 1 yarn hadoop 138292 Dec 13 2019 ./tezlib/lib/hadoop-auth-3.1.1.3.1.5.0-152.jar 6443032003 296 -r-xr-xr-x 1 yarn hadoop 299508 Dec 5 2019 ./tezlib/lib/nimbus-jose-jwt-4.41.1.jar 6443032010 8 -r-xr-xr-x 1 yarn hadoop 4722 Oct 22 2016 ./tezlib/lib/jcip-annotations-1.0-1.jar 6443032011 120 -r-xr-xr-x 1 yarn hadoop 120316 Dec 5 2019 ./tezlib/lib/json-smart-2.3.jar 6443032032 32 -r-xr-xr-x 1 yarn hadoop 30035 Dec 5 2019 ./tezlib/lib/accessors-smart-1.2.jar 6443032033 56 -r-xr-xr-x 1 yarn hadoop 53297 Oct 22 2016 ./tezlib/lib/asm-5.0.4.jar 6443032034 824 -r-xr-xr-x 1 yarn hadoop 839856 Dec 13 2019 ./tezlib/lib/zookeeper-3.4.6.3.1.5.0-152.jar 6443032035 1300 -r-xr-xr-x 1 yarn hadoop 1330394 Oct 22 2016 ./tezlib/lib/netty-3.10.5.Final.jar 6443032036 200 -r-xr-xr-x 1 yarn hadoop 201953 Dec 5 2019 ./tezlib/lib/curator-framework-2.12.0.jar 6443032037 2368 -r-xr-xr-x 1 yarn hadoop 2423138 Dec 5 2019 ./tezlib/lib/curator-client-2.12.0.jar 6443032042 20 -r-xr-xr-x 1 yarn hadoop 20409 Dec 5 2019 ./tezlib/lib/kerb-simplekdc-1.0.1.jar 6443032043 112 -r-xr-xr-x 1 yarn hadoop 113017 Dec 5 2019 ./tezlib/lib/kerb-client-1.0.1.jar 6443032044 32 -r-xr-xr-x 1 yarn hadoop 30674 Dec 5 2019 ./tezlib/lib/kerby-config-1.0.1.jar 6443032045 224 -r-xr-xr-x 1 yarn hadoop 226672 Dec 5 2019 ./tezlib/lib/kerb-core-1.0.1.jar 6443032046 200 -r-xr-xr-x 1 yarn hadoop 204650 Dec 5 2019 ./tezlib/lib/kerby-pkix-1.0.1.jar 6443032047 100 -r-xr-xr-x 1 yarn hadoop 102174 Dec 5 2019 ./tezlib/lib/kerby-asn1-1.0.1.jar 6443032048 40 -r-xr-xr-x 1 yarn hadoop 40554 Dec 5 2019 ./tezlib/lib/kerby-util-1.0.1.jar 6443032049 64 -r-xr-xr-x 1 yarn hadoop 65464 Dec 5 2019 ./tezlib/lib/kerb-common-1.0.1.jar 6443032050 116 -r-xr-xr-x 1 yarn hadoop 116120 Dec 5 2019 ./tezlib/lib/kerb-crypto-1.0.1.jar 6443032052 36 -r-xr-xr-x 1 yarn hadoop 36708 Dec 5 2019 ./tezlib/lib/kerb-util-1.0.1.jar 6443032053 20 -r-xr-xr-x 1 yarn hadoop 18763 Dec 5 2019 ./tezlib/lib/token-provider-1.0.1.jar 6443032056 80 -r-xr-xr-x 1 yarn hadoop 80980 Dec 5 2019 ./tezlib/lib/kerb-admin-1.0.1.jar 6443032057 84 -r-xr-xr-x 1 yarn hadoop 82756 Dec 5 2019 ./tezlib/lib/kerb-server-1.0.1.jar 6443032060 20 -r-xr-xr-x 1 yarn hadoop 20046 Dec 5 2019 ./tezlib/lib/kerb-identity-1.0.1.jar 6443032061 32 -r-xr-xr-x 1 yarn hadoop 29134 Dec 5 2019 ./tezlib/lib/kerby-xdr-1.0.1.jar 6443122018 280 -r-xr-xr-x 1 yarn hadoop 283598 Dec 5 2019 ./tezlib/lib/curator-recipes-2.12.0.jar 6443122019 1468 -r-xr-xr-x 1 yarn hadoop 1502280 Dec 5 2019 ./tezlib/lib/htrace-core4-4.1.0-incubating.jar 6443122021 88 -r-xr-xr-x 1 yarn hadoop 87325 Oct 22 2016 ./tezlib/lib/jline-0.9.94.jar 6443122022 1372 -r-xr-xr-x 1 yarn hadoop 1400944 Dec 6 2019 ./tezlib/lib/jackson-databind-2.10.0.jar 6443122023 344 -r-xr-xr-x 1 yarn hadoop 348635 Dec 6 2019 ./tezlib/lib/jackson-core-2.10.0.jar 6443122024 160 -r-xr-xr-x 1 yarn hadoop 161867 Oct 22 2016 ./tezlib/lib/stax2-api-3.1.4.jar 6443122025 504 -r-xr-xr-x 1 yarn hadoop 512742 Dec 5 2019 ./tezlib/lib/woodstox-core-5.0.3.jar 6443122028 464 -r-xr-xr-x 1 yarn hadoop 473316 Dec 13 2019 ./tezlib/lib/hadoop-aws-3.1.1.3.1.5.0-152.jar 6443122029 96420 -r-xr-xr-x 1 yarn hadoop 98732349 Dec 6 2019 ./tezlib/lib/aws-java-sdk-bundle-1.11.375.jar 6443122072 488 -r-xr-xr-x 1 yarn hadoop 497322 Dec 13 2019 ./tezlib/lib/hadoop-azure-3.1.1.3.1.5.0-152.jar 6443122073 796 -r-xr-xr-x 1 yarn hadoop 812977 Dec 6 2019 ./tezlib/lib/azure-storage-7.0.0.jar 6443122075 12 -r-xr-xr-x 1 yarn hadoop 10288 Dec 6 2019 ./tezlib/lib/azure-keyvault-core-1.0.0.jar 6443181130 40 -r-xr-xr-x 1 yarn hadoop 38526 Dec 5 2019 ./tezlib/lib/jetty-util-ajax-9.3.24.v20180605.jar 6443181131 448 -r-xr-xr-x 1 yarn hadoop 458642 Dec 5 2019 ./tezlib/lib/jetty-util-9.3.24.v20180605.jar 6443181132 416 -r-xr-xr-x 1 yarn hadoop 424076 Dec 6 2019 ./tezlib/lib/wildfly-openssl-1.0.7.Final.jar 6443181133 68 -r-xr-xr-x 1 yarn hadoop 67086 Dec 13 2019 ./tezlib/lib/hadoop-azure-datalake-3.1.1.3.1.5.0-152.jar 6443181136 512 -r-xr-xr-x 1 yarn hadoop 520682 Dec 6 2019 ./tezlib/lib/azure-data-lake-store-sdk-2.3.3.jar 6443181137 4048 -r-xr-xr-x 1 yarn hadoop 4141576 Dec 13 2019 ./tezlib/lib/gcs-connector-hadoop3-1.9.17.3.1.5.0-152-shaded.jar 6443181139 8 -r-xr-xr-x 1 yarn hadoop 6849 Dec 13 2019 ./tezlib/lib/google-extensions-0.3.1.jar 6443181150 100 -r-xr-xr-x 1 yarn hadoop 100356 Dec 13 2019 ./tezlib/lib/flogger-0.3.1.jar 6443181151 16 -r-xr-xr-x 1 yarn hadoop 15388 Dec 13 2019 ./tezlib/lib/flogger-system-backend-0.3.1.jar 6443181152 8 -r-xr-xr-x 1 yarn hadoop 7586 Dec 13 2019 ./tezlib/lib/flogger-log4j-backend-0.3.1.jar 6443181153 28 -r-xr-xr-x 1 yarn hadoop 26514 Oct 22 2016 ./tezlib/lib/stax-api-1.0.1.jar 6443181154 2776 -r-xr-xr-x 1 yarn hadoop 2839848 Dec 13 2019 ./tezlib/lib/hadoop-yarn-common-3.1.1.3.1.5.0-152.jar 6443181155 428 -r-xr-xr-x 1 yarn hadoop 436689 Oct 22 2016 ./tezlib/lib/jersey-core-1.19.jar 6443181158 48 -r-xr-xr-x 1 yarn hadoop 46367 Oct 22 2016 ./tezlib/lib/jsr311-api-1.1.1.jar 6443181159 132 -r-xr-xr-x 1 yarn hadoop 134021 Oct 22 2016 ./tezlib/lib/jersey-client-1.19.jar 6443181164 76 -r-xr-xr-x 1 yarn hadoop 76983 Dec 13 2019 ./tezlib/lib/guice-servlet-4.0.jar 6443181165 656 -r-xr-xr-x 1 yarn hadoop 668235 Dec 13 2019 ./tezlib/lib/guice-4.0.jar 6443181180 4 -r-xr-xr-x 1 yarn hadoop 2497 Oct 22 2016 ./tezlib/lib/javax.inject-1.jar 6443181181 8 -r-xr-xr-x 1 yarn hadoop 4467 Oct 22 2016 ./tezlib/lib/aopalliance-1.0.jar 6443181182 16 -r-xr-xr-x 1 yarn hadoop 16151 Oct 22 2016 ./tezlib/lib/jersey-guice-1.19.jar 6443181183 36 -r-xr-xr-x 1 yarn hadoop 34991 Dec 6 2019 ./tezlib/lib/jackson-module-jaxb-annotations-2.10.0.jar 6443210306 116 -r-xr-xr-x 1 yarn hadoop 115498 Dec 6 2019 ./tezlib/lib/jakarta.xml.bind-api-2.3.2.jar 6443210307 44 -r-xr-xr-x 1 yarn hadoop 44399 Dec 6 2019 ./tezlib/lib/jakarta.activation-api-1.2.1.jar 6443210312 16 -r-xr-xr-x 1 yarn hadoop 15529 Dec 6 2019 ./tezlib/lib/jackson-jaxrs-json-provider-2.10.0.jar 6443210313 36 -r-xr-xr-x 1 yarn hadoop 33176 Dec 6 2019 ./tezlib/lib/jackson-jaxrs-base-2.10.0.jar 6443210346 300 -r-xr-xr-x 1 yarn hadoop 303191 Dec 13 2019 ./tezlib/lib/hadoop-yarn-client-3.1.1.3.1.5.0-152.jar 6443210347 736 -r-xr-xr-x 1 yarn hadoop 751238 Oct 22 2016 ./tezlib/lib/commons-collections4-4.1.jar 6443210350 512 -r-xr-xr-x 1 yarn hadoop 520813 Dec 5 2019 ./tezlib/lib/jetty-server-9.3.24.v20180605.jar 6443210351 148 -r-xr-xr-x 1 yarn hadoop 150638 Dec 5 2019 ./tezlib/lib/jetty-http-9.3.24.v20180605.jar 6443210352 120 -r-xr-xr-x 1 yarn hadoop 122669 Dec 5 2019 ./tezlib/lib/jetty-io-9.3.24.v20180605.jar 6443210353 24 -r-xr-xr-x 1 yarn hadoop 24239 Oct 22 2016 ./tezlib/lib/commons-daemon-1.0.13.jar 6443210358 1024 -r-xr-xr-x 1 yarn hadoop 1045744 Oct 22 2016 ./tezlib/lib/leveldbjni-all-1.8.jar 6443210359 4872 -r-xr-xr-x 1 yarn hadoop 4986447 Dec 13 2019 ./tezlib/lib/hadoop-hdfs-client-3.1.1.3.1.5.0-152.jar 6443210365 324 -r-xr-xr-x 1 yarn hadoop 331034 Dec 5 2019 ./tezlib/lib/okhttp-2.7.5.jar 6443699392 68 -r-xr-xr-x 1 yarn hadoop 65928 Dec 5 2019 ./tezlib/lib/okio-1.6.0.jar 6443699393 124 -r-xr-xr-x 1 yarn hadoop 124846 Oct 22 2016 ./tezlib/lib/RoaringBitmap-0.4.9.jar 6443699394 728 -r-xr-xr-x 1 yarn hadoop 745264 Dec 6 2019 ./tezlib/lib/async-http-client-1.9.40.jar 6443699395 1620 -r-xr-xr-x 1 yarn hadoop 1654823 Dec 13 2019 ./tezlib/lib/hadoop-mapreduce-client-core-3.1.1.3.1.5.0-152.jar 6443699396 788 -r-xr-xr-x 1 yarn hadoop 803565 Dec 13 2019 ./tezlib/lib/hadoop-mapreduce-client-common-3.1.1.3.1.5.0-152.jar 6443699397 80 -r-xr-xr-x 1 yarn hadoop 80533 Dec 13 2019 ./tezlib/lib/hadoop-yarn-server-web-proxy-3.1.1.3.1.5.0-152.jar 6443699398 1308 -r-xr-xr-x 1 yarn hadoop 1336863 Dec 13 2019 ./tezlib/lib/hadoop-yarn-server-common-3.1.1.3.1.5.0-152.jar 6443699399 224 -r-xr-xr-x 1 yarn hadoop 225702 Dec 13 2019 ./tezlib/lib/hadoop-yarn-registry-3.1.1.3.1.5.0-152.jar 6443699400 304 -r-xr-xr-x 1 yarn hadoop 307637 Oct 23 2016 ./tezlib/lib/dnsjava-2.1.7.jar 6443699401 56 -r-xr-xr-x 1 yarn hadoop 55236 Dec 5 2019 ./tezlib/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar 6443699412 1688 -r-xr-xr-x 1 yarn hadoop 1726527 Dec 5 2019 ./tezlib/lib/ehcache-3.3.1.jar 6443699413 132 -r-xr-xr-x 1 yarn hadoop 134308 Dec 5 2019 ./tezlib/lib/HikariCP-java7-2.4.12.jar 6443699414 776 -r-xr-xr-x 1 yarn hadoop 792442 Dec 5 2019 ./tezlib/lib/mssql-jdbc-6.2.1.jre7.jar 6443699415 4092 -r-xr-xr-x 1 yarn hadoop 4189874 Dec 6 2019 ./tezlib/lib/bcprov-jdk15on-1.60.jar 6443699416 780 -r-xr-xr-x 1 yarn hadoop 796532 Dec 6 2019 ./tezlib/lib/bcpkix-jdk15on-1.60.jar 6443699417 104 -r-xr-xr-x 1 yarn hadoop 105112 Oct 22 2016 ./tezlib/lib/servlet-api-2.5.jar 6443699424 112 -r-xr-xr-x 1 yarn hadoop 111908 Oct 22 2016 ./tezlib/lib/metrics-core-3.1.0.jar 6443699425 288 -r-xr-xr-x 1 yarn hadoop 294844 Dec 13 2019 ./tezlib/lib/hadoop-yarn-server-applicationhistoryservice-3.1.1.3.1.5.0-152.jar 6443699426 28 -r-xr-xr-x 1 yarn hadoop 28569 Oct 22 2016 ./tezlib/lib/objenesis-1.0.jar 6443699427 380 -r-xr-xr-x 1 yarn hadoop 387689 Dec 5 2019 ./tezlib/lib/fst-2.50.jar 6443699428 60 -r-xr-xr-x 1 yarn hadoop 58487 Dec 5 2019 ./tezlib/lib/java-util-1.9.0.jar 6443699429 76 -r-xr-xr-x 1 yarn hadoop 75232 Dec 5 2019 ./tezlib/lib/json-io-2.5.1.jar 6443699430 92 -r-xr-xr-x 1 yarn hadoop 93196 Dec 13 2019 ./tezlib/lib/hadoop-yarn-server-timeline-pluginstorage-3.1.1.3.1.5.0-152.jar 6443699431 4 drwxr-xr-x 3 yarn hadoop 4096 Dec 13 2019 ./tezlib/lib/native 6443699432 1784 -r-xr-xr-x 1 yarn hadoop 1824978 Dec 13 2019 ./tezlib/lib/native/libhadoop.a 6443699433 3324 -r-xr-xr-x 2 yarn hadoop 3402313 Dec 13 2019 ./tezlib/lib/native/libnativetask.so.1.0.0 6443699434 24 -r-xr-xr-x 3 yarn hadoop 23800 Jun 10 2014 ./tezlib/lib/native/libsnappy.so.1.1.4 6443699435 8396 -r-xr-xr-x 1 yarn hadoop 8594282 Dec 13 2019 ./tezlib/lib/native/libnativetask.a 6443699434 24 -r-xr-xr-x 3 yarn hadoop 23800 Jun 10 2014 ./tezlib/lib/native/libsnappy.so.1 6443726145 1008 -r-xr-xr-x 2 yarn hadoop 1029214 Dec 13 2019 ./tezlib/lib/native/libhadoop.so.1.0.0 6443726147 452 -r-xr-xr-x 1 yarn hadoop 459400 Dec 13 2019 ./tezlib/lib/native/libhdfs.a 6443726152 280 -r-xr-xr-x 2 yarn hadoop 286676 Dec 13 2019 ./tezlib/lib/native/libhdfs.so 6443726145 1008 -r-xr-xr-x 2 yarn hadoop 1029214 Dec 13 2019 ./tezlib/lib/native/libhadoop.so 6443699434 24 -r-xr-xr-x 3 yarn hadoop 23800 Jun 10 2014 ./tezlib/lib/native/libsnappy.so 6443726153 468 -r-xr-xr-x 1 yarn hadoop 476136 Dec 13 2019 ./tezlib/lib/native/libhadooputils.a 6443726156 0 drwxr-xr-x 2 yarn hadoop 114 Dec 13 2019 ./tezlib/lib/native/examples 6443726157 904 -r-xr-xr-x 1 yarn hadoop 924089 Dec 13 2019 ./tezlib/lib/native/examples/pipes-sort 6443726160 924 -r-xr-xr-x 1 yarn hadoop 943052 Dec 13 2019 ./tezlib/lib/native/examples/wordcount-part 6443726161 920 -r-xr-xr-x 1 yarn hadoop 938380 Dec 13 2019 ./tezlib/lib/native/examples/wordcount-simple 6443726162 964 -r-xr-xr-x 1 yarn hadoop 983078 Dec 13 2019 ./tezlib/lib/native/examples/wordcount-nopipe 6443726163 1572 -r-xr-xr-x 1 yarn hadoop 1607384 Dec 13 2019 ./tezlib/lib/native/libhadooppipes.a 6443726152 280 -r-xr-xr-x 2 yarn hadoop 286676 Dec 13 2019 ./tezlib/lib/native/libhdfs.so.0.0.0 6443699433 3324 -r-xr-xr-x 2 yarn hadoop 3402313 Dec 13 2019 ./tezlib/lib/native/libnativetask.so 2147484452 41784 -r-x------ 1 yarn hadoop 42785708 Dec 4 15:33 ./hive-exec-3.1.0.3.1.5.0-152-52ac635d1c2eaf5d3e864f9c6dd1cc6c640a4133a03a6747e20ad2156829ebdf.jar 2149762116 264 -r-x------ 1 yarn hadoop 269276 Jan 29 13:32 ./hive-hcatalog-core.jar 2150202189 128 -r-x------ 1 yarn hadoop 127507 Jan 29 13:32 ./tez-conf.pb broken symlinks(find -L . -maxdepth 5 -type l -ls): End of LogType:directory.info ******************************************************************************* Container: container_e703_1611750110647_64528_01_000001 on wx12-dsj-node045_45454_1611898349701 LogAggregationType: AGGREGATED =============================================================================================== LogType:launch_container.sh LogLastModifiedTime:Fri Jan 29 13:32:30 +0800 2021 LogLength:5702 LogContents: #!/bin/bash set -o pipefail -e export PRELAUNCH_OUT="/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/prelaunch.out" exec >"${PRELAUNCH_OUT}" export PRELAUNCH_ERR="/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/prelaunch.err" exec 2>"${PRELAUNCH_ERR}" echo "Setting up env variables" export JAVA_HOME=${JAVA_HOME:-"/usr/jdk64/jdk1.8.0_112"} export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/hdp/3.1.5.0-152/hadoop/conf"} export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/hdp/3.1.5.0-152/hadoop-yarn"} export HADOOP_HOME=${HADOOP_HOME:-"/usr/hdp/3.1.5.0-152/hadoop"} export PATH=${PATH:-"/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/jdk64/jdk1.8.0_112/bin:/usr/hdp/3.1.5.0-152/flink/bin:/opt/app/datax/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent"} export LANG=${LANG:-"en_US.UTF-8"} export HADOOP_TOKEN_FILE_LOCATION="/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/container_tokens" export CONTAINER_ID="container_e703_1611750110647_64528_01_000001" export NM_PORT="45454" export NM_HOST="wx12-dsj-node045" export NM_HTTP_PORT="8042" export LOCAL_DIRS="/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528" export LOCAL_USER_DIRS="/data01/hdp/yarn/local/usercache/data_quality/" export LOG_DIRS="/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001" export USER="data_quality" export LOGNAME="data_quality" export HOME="/home/" export PWD="/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001" export JVM_PID="$$" export MALLOC_ARENA_MAX="4" export NM_AUX_SERVICE_timeline_collector="" export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=" export NM_AUX_SERVICE_spark2_shuffle="" export TEZ_CLIENT_VERSION="0.9.1.3.1.5.0-152" export APP_SUBMIT_TIME_ENV="1611898322753" export TIMELINE_FLOW_NAME_TAG="HIVE-74300ba4-ffcf-43e7-87c4-a2ea8a280d1c" export LD_LIBRARY_PATH="$PWD:/usr/hdp/3.1.5.0-152/hadoop/lib/native:/usr/hdp/3.1.5.0-152/hadoop/lib/native/Linux-amd64-64:$PWD:/usr/hdp/3.1.5.0-152/hadoop/lib/native:/usr/hdp/3.1.5.0-152/hadoop/lib/native/Linux-amd64-64:$HADOOP_COMMON_HOME/lib/native/" export TIMELINE_FLOW_VERSION_TAG="1" export APPLICATION_WEB_PROXY_BASE="/proxy/application_1611750110647_64528" export CLASSPATH="/usr/hdp/3.1.5.0-152/hadoop/lib/hadoop-lzo-0.6.0.3.1.5.0-152.jar:/etc/hadoop/conf/secure:$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*:" export TIMELINE_FLOW_RUN_ID_TAG="1611898322754" echo "Setting up job resources" ln -sf "/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/filecache/10/tez.session.local-resources.pb" "tez.session.local-resources.pb" ln -sf "/data01/hdp/yarn/local/filecache/2160/tez.tar.gz" "tezlib" ln -sf "/data01/hdp/yarn/local/usercache/data_quality/filecache/19219/hive-exec-3.1.0.3.1.5.0-152-52ac635d1c2eaf5d3e864f9c6dd1cc6c640a4133a03a6747e20ad2156829ebdf.jar" "hive-exec-3.1.0.3.1.5.0-152-52ac635d1c2eaf5d3e864f9c6dd1cc6c640a4133a03a6747e20ad2156829ebdf.jar" ln -sf "/data01/hdp/yarn/local/usercache/data_quality/filecache/45853/hive-hcatalog-core.jar" "hive-hcatalog-core.jar" ln -sf "/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/filecache/11/tez-conf.pb" "tez-conf.pb" echo "Copying debugging information" # Creating copy of launch script cp "launch_container.sh" "/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/launch_container.sh" chmod 640 "/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/launch_container.sh" # Determining directory contents echo "ls -l:" 1>"/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/directory.info" ls -l 1>>"/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/directory.info" echo "find -L . -maxdepth 5 -ls:" 1>>"/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/directory.info" find -L . -maxdepth 5 -ls 1>>"/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/directory.info" echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/directory.info" find -L . -maxdepth 5 -type l -ls 1>>"/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/directory.info" echo "Launching container" exec /bin/bash -c "$JAVA_HOME/bin/java -Xmx3276m -Djava.io.tmpdir=$PWD/tmp -server -Djava.net.preferIPv4Stack=true -Dhdp.version=3.1.5.0-152 -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster --session 1>/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/stdout 2>/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/stderr " End of LogType:launch_container.sh ************************************************************************************ End of LogType:prelaunch.err ****************************************************************************** Container: container_e703_1611750110647_64528_01_000001 on wx12-dsj-node045_45454_1611898349701 LogAggregationType: AGGREGATED =============================================================================================== LogType:prelaunch.out LogLastModifiedTime:Fri Jan 29 13:32:30 +0800 2021 LogLength:100 LogContents: Setting up env variables Setting up job resources Copying debugging information Launching container End of LogType:prelaunch.out ****************************************************************************** Container: container_e703_1611750110647_64528_01_000001 on wx12-dsj-node045_45454_1611898349701 LogAggregationType: AGGREGATED =============================================================================================== LogType:stderr LogLastModifiedTime:Fri Jan 29 13:32:30 +0800 2021 LogLength:120 LogContents: 2021-01-29 13:32:10 Running Dag: dag_1611750110647_64528_1 2021-01-29 13:32:20 Completed Dag: dag_1611750110647_64528_1 End of LogType:stderr *********************************************************************** Container: container_e703_1611750110647_64528_01_000001 on wx12-dsj-node045_45454_1611898349701 LogAggregationType: AGGREGATED =============================================================================================== LogType:stdout LogLastModifiedTime:Fri Jan 29 13:32:30 +0800 2021 LogLength:7561 LogContents: 1.406: [GC pause (G1 Evacuation Pause) (young), 0.0352248 secs] [Parallel Time: 22.0 ms, GC Workers: 43] [GC Worker Start (ms): Min: 1406.2, Avg: 1413.2, Max: 1427.5, Diff: 21.3] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 4.1, Diff: 4.1, Sum: 12.3] [Update RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] [Processed Buffers: Min: 0, Avg: 0.0, Max: 0, Diff: 0, Sum: 0] [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.6] [Code Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 4.2, Diff: 4.2, Sum: 14.8] [Object Copy (ms): Min: 0.0, Avg: 8.7, Max: 14.2, Diff: 14.2, Sum: 374.0] [Termination (ms): Min: 0.0, Avg: 5.0, Max: 6.0, Diff: 6.0, Sum: 215.1] [Termination Attempts: Min: 1, Avg: 35.3, Max: 62, Diff: 61, Sum: 1520] [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.0, Sum: 0.8] [GC Worker Total (ms): Min: 0.0, Avg: 14.4, Max: 21.4, Diff: 21.4, Sum: 617.7] [GC Worker End (ms): Min: 1427.5, Avg: 1427.6, Max: 1427.7, Diff: 0.2] [Code Root Fixup: 0.7 ms] [Code Root Purge: 0.1 ms] [Clear CT: 1.0 ms] [Other: 11.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 8.7 ms] [Ref Enq: 0.1 ms] [Redirty Cards: 1.0 ms] [Humongous Register: 0.2 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.0 ms] [Eden: 102.0M(102.0M)->0.0B(89.0M) Survivors: 0.0B->13.0M Heap: 102.0M(2048.0M)->13.4M(2048.0M)] [Times: user=0.60 sys=0.05, real=0.03 secs] 1.738: [GC pause (Metadata GC Threshold) (young) (initial-mark), 0.0244476 secs] [Parallel Time: 16.3 ms, GC Workers: 43] [GC Worker Start (ms): Min: 1737.9, Avg: 1738.4, Max: 1739.0, Diff: 1.1] [Ext Root Scanning (ms): Min: 1.8, Avg: 2.7, Max: 8.3, Diff: 6.5, Sum: 116.9] [Update RS (ms): Min: 0.0, Avg: 0.0, Max: 0.2, Diff: 0.2, Sum: 0.2] [Processed Buffers: Min: 0, Avg: 0.0, Max: 1, Diff: 1, Sum: 1] [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.6] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 5.5, Diff: 5.5, Sum: 6.3] [Object Copy (ms): Min: 6.7, Avg: 11.6, Max: 12.6, Diff: 5.9, Sum: 497.3] [Termination (ms): Min: 0.0, Avg: 0.5, Max: 0.6, Diff: 0.6, Sum: 22.3] [Termination Attempts: Min: 1, Avg: 36.1, Max: 52, Diff: 51, Sum: 1553] [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.9] [GC Worker Total (ms): Min: 14.3, Avg: 15.0, Max: 15.5, Diff: 1.2, Sum: 644.6] [GC Worker End (ms): Min: 1753.3, Avg: 1753.4, Max: 1753.5, Diff: 0.2] [Code Root Fixup: 0.8 ms] [Code Root Purge: 0.1 ms] [Clear CT: 1.0 ms] [Other: 6.3 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 1.0 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 0.3 ms] [Eden: 79.0M(89.0M)->0.0B(120.0M) Survivors: 13.0M->13.0M Heap: 91.9M(2048.0M)->16.9M(2048.0M)] [Times: user=0.56 sys=0.01, real=0.02 secs] 1.762: [GC concurrent-root-region-scan-start] 1.766: [GC concurrent-root-region-scan-end, 0.0033128 secs] 1.766: [GC concurrent-mark-start] 1.768: [GC concurrent-mark-end, 0.0020323 secs] 1.768: [GC remark 1.768: [Finalize Marking, 0.0037054 secs] 1.772: [GC ref-proc, 0.0026198 secs] 1.774: [Unloading, 0.0069220 secs], 0.0154630 secs] [Times: user=0.26 sys=0.01, real=0.01 secs] 1.784: [GC cleanup 17M->17M(2048M), 0.0087668 secs] [Times: user=0.09 sys=0.06, real=0.00 secs] 2.505: [GC pause (G1 Evacuation Pause) (young), 0.0590618 secs] [Parallel Time: 45.8 ms, GC Workers: 43] [GC Worker Start (ms): Min: 2505.0, Avg: 2505.6, Max: 2506.5, Diff: 1.5] [Ext Root Scanning (ms): Min: 0.0, Avg: 1.4, Max: 32.1, Diff: 32.0, Sum: 61.6] [Update RS (ms): Min: 0.0, Avg: 0.0, Max: 1.2, Diff: 1.2, Sum: 2.0] [Processed Buffers: Min: 0, Avg: 0.1, Max: 1, Diff: 1, Sum: 4] [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.8] [Code Root Scanning (ms): Min: 0.0, Avg: 3.3, Max: 37.8, Diff: 37.8, Sum: 143.0] [Object Copy (ms): Min: 6.0, Avg: 39.7, Max: 43.9, Diff: 37.9, Sum: 1706.8] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 4.5] [Termination Attempts: Min: 1, Avg: 57.1, Max: 90, Diff: 89, Sum: 2455] [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 6.5] [GC Worker Total (ms): Min: 43.7, Avg: 44.8, Max: 45.5, Diff: 1.7, Sum: 1925.2] [GC Worker End (ms): Min: 2550.3, Avg: 2550.4, Max: 2550.6, Diff: 0.4] [Code Root Fixup: 1.4 ms] [Code Root Purge: 0.0 ms] [Clear CT: 1.1 ms] [Other: 10.8 ms] [Choose CSet: 0.0 ms] [Ref Proc: 8.7 ms] [Ref Enq: 0.1 ms] [Redirty Cards: 1.0 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 0.4 ms] [Eden: 120.0M(120.0M)->0.0B(583.0M) Survivors: 13.0M->17.0M Heap: 136.9M(2048.0M)->27.5M(2048.0M)] [Times: user=0.72 sys=0.30, real=0.06 secs] 5.444: [GC pause (Metadata GC Threshold) (young) (initial-mark), 0.0357006 secs] [Parallel Time: 27.0 ms, GC Workers: 43] [GC Worker Start (ms): Min: 5444.6, Avg: 5445.1, Max: 5445.6, Diff: 1.0] [Ext Root Scanning (ms): Min: 1.8, Avg: 2.6, Max: 3.4, Diff: 1.6, Sum: 109.8] [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.8, Diff: 0.8, Sum: 4.9] [Processed Buffers: Min: 0, Avg: 0.2, Max: 2, Diff: 2, Sum: 10] [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.0, Sum: 0.7] [Code Root Scanning (ms): Min: 0.0, Avg: 0.2, Max: 5.3, Diff: 5.3, Sum: 8.9] [Object Copy (ms): Min: 17.5, Avg: 22.5, Max: 23.0, Diff: 5.5, Sum: 967.4] [Termination (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.2, Sum: 9.5] [Termination Attempts: Min: 1, Avg: 111.5, Max: 136, Diff: 135, Sum: 4793] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.8] [GC Worker Total (ms): Min: 25.1, Avg: 25.7, Max: 26.1, Diff: 1.0, Sum: 1104.1] [GC Worker End (ms): Min: 5470.7, Avg: 5470.8, Max: 5470.9, Diff: 0.2] [Code Root Fixup: 1.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.8 ms] [Other: 6.8 ms] [Choose CSet: 0.0 ms] [Ref Proc: 2.9 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 2.7 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 0.2 ms] [Eden: 66.0M(582.0M)->0.0B(878.0M) Survivors: 17.0M->25.0M Heap: 93.5M(2048.0M)->34.7M(2048.0M)] [Times: user=1.03 sys=0.01, real=0.03 secs] 5.480: [GC concurrent-root-region-scan-start] 5.485: [GC concurrent-root-region-scan-end, 0.0050434 secs] 5.485: [GC concurrent-mark-start] 5.487: [GC concurrent-mark-end, 0.0017661 secs] 5.488: [GC remark 5.488: [Finalize Marking, 0.0048080 secs] 5.492: [GC ref-proc, 0.0010193 secs] 5.493: [Unloading, 0.0071879 secs], 0.0143212 secs] [Times: user=0.30 sys=0.00, real=0.01 secs] 5.502: [GC cleanup 35M->24M(2048M), 0.0062395 secs] [Times: user=0.09 sys=0.00, real=0.01 secs] 5.508: [GC concurrent-cleanup-start] 5.508: [GC concurrent-cleanup-end, 0.0000423 secs] 2021-01-29 13:32:10 Running Dag: dag_1611750110647_64528_1 2021-01-29 13:32:20 Completed Dag: dag_1611750110647_64528_1 Heap garbage-first heap total 2097152K, used 189684K [0x00000006f3400000, 0x00000006f3504000, 0x00000007c0000000) region size 1024K, 187 young (191488K), 25 survivors (25600K) Metaspace used 50988K, capacity 51348K, committed 51840K, reserved 1095680K class space used 5869K, capacity 5972K, committed 6016K, reserved 1048576K End of LogType:stdout *********************************************************************** Container: container_e703_1611750110647_64528_01_000001 on wx12-dsj-node045_45454_1611898349701 LogAggregationType: AGGREGATED =============================================================================================== LogType:syslog LogLastModifiedTime:Fri Jan 29 13:32:30 +0800 2021 LogLength:44325 LogContents: 2021-01-29 13:32:05,729 [INFO] [main] |app.DAGAppMaster|: Creating DAGAppMaster for applicationId=application_1611750110647_64528, attemptNum=1, AMContainerId=container_e703_1611750110647_64528_01_000001, jvmPid=11567, userFromEnv=data_quality, cliSessionOption=true, pwd=/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001, localDirs=/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528, logDirs=/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001 2021-01-29 13:32:05,983 [INFO] [main] |security.SecurityUtil|: Updating Configuration 2021-01-29 13:32:06,138 [INFO] [main] |app.DAGAppMaster|: Created DAGAppMaster for application appattempt_1611750110647_64528_000001, versionInfo=[ component=tez-dag, version=0.9.1.3.1.5.0-152, revision=65e86f7706bc6441890e78f003c8e89abbe5a79f, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=2019-12-12T20:16:29Z ] 2021-01-29 13:32:06,205 [INFO] [main] |app.DAGAppMaster|: /************************************************************ [system properties] os.name: Linux os.version: 3.10.0-1127.el7.x86_64 java.home: /usr/jdk64/jdk1.8.0_112/jre java.runtime.version: 1.8.0_112-b15 java.vendor: Oracle Corporation java.version: 1.8.0_112 java.vm.name: Java HotSpot(TM) 64-Bit Server VM java.class.path: /usr/hdp/3.1.5.0-152/hadoop/lib/hadoop-lzo-0.6.0.3.1.5.0-152.jar:/etc/hadoop/conf/secure:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/hive-exec-3.1.0.3.1.5.0-152-52ac635d1c2eaf5d3e864f9c6dd1cc6c640a4133a03a6747e20ad2156829ebdf.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/hive-hcatalog-core.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/hadoop-shim-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-api-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-common-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-runtime-internals-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-runtime-library-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-mapreduce-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-examples-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-dag-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-tests-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-ext-service-tests-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-protobuf-history-plugin-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-yarn-timeline-history-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-yarn-timeline-history-with-acls-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-yarn-timeline-cache-plugin-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-yarn-timeline-history-with-fs-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-history-parser-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-job-analyzer-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/tez-javadoc-tools-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/hadoop-shim-2.8-0.9.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/slf4j-api-1.7.10.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-yarn-api-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/commons-lang-2.6.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/guava-28.0-jre.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/failureaccess-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jsr305-3.0.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/checker-qual-2.8.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/error_prone_annotations-2.3.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/j2objc-annotations-1.3.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/animal-sniffer-annotations-1.17.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/commons-logging-1.1.3.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jaxb-api-2.2.11.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/protobuf-java-2.5.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jackson-annotations-2.10.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-common-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-annotations-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/commons-cli-1.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/commons-math3-3.1.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/httpclient-4.5.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/httpcore-4.4.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/commons-codec-1.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/commons-io-2.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/commons-net-3.6.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/commons-collections-3.2.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/javax.servlet-api-3.1.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jetty-servlet-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jetty-security-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jetty-webapp-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jetty-xml-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jersey-servlet-1.19.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jersey-json-1.19.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jettison-1.3.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jaxb-impl-2.2.3-1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jackson-core-asl-1.9.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jackson-mapper-asl-1.9.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jackson-jaxrs-1.9.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jackson-xc-1.9.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/log4j-1.2.17.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/commons-beanutils-1.9.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/commons-configuration2-2.1.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/commons-lang3-3.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/slf4j-log4j12-1.7.10.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/avro-1.7.7.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/paranamer-2.3.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/snappy-java-1.0.5.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/commons-compress-1.4.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/xz-1.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/re2j-1.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/gson-2.2.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-auth-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/nimbus-jose-jwt-4.41.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jcip-annotations-1.0-1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/json-smart-2.3.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/accessors-smart-1.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/asm-5.0.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/zookeeper-3.4.6.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/netty-3.10.5.Final.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/curator-framework-2.12.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/curator-client-2.12.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/kerb-simplekdc-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/kerb-client-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/kerby-config-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/kerb-core-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/kerby-pkix-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/kerby-asn1-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/kerby-util-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/kerb-common-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/kerb-crypto-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/kerb-util-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/token-provider-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/kerb-admin-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/kerb-server-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/kerb-identity-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/kerby-xdr-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/curator-recipes-2.12.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/htrace-core4-4.1.0-incubating.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jline-0.9.94.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jackson-databind-2.10.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jackson-core-2.10.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/stax2-api-3.1.4.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/woodstox-core-5.0.3.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-aws-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/aws-java-sdk-bundle-1.11.375.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-azure-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/azure-storage-7.0.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/azure-keyvault-core-1.0.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jetty-util-ajax-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jetty-util-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/wildfly-openssl-1.0.7.Final.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-azure-datalake-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/azure-data-lake-store-sdk-2.3.3.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/gcs-connector-hadoop3-1.9.17.3.1.5.0-152-shaded.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/google-extensions-0.3.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/flogger-0.3.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/flogger-system-backend-0.3.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/flogger-log4j-backend-0.3.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/stax-api-1.0.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-yarn-common-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jersey-core-1.19.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jsr311-api-1.1.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jersey-client-1.19.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/guice-servlet-4.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/guice-4.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/javax.inject-1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/aopalliance-1.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jersey-guice-1.19.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jackson-module-jaxb-annotations-2.10.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jakarta.xml.bind-api-2.3.2.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jakarta.activation-api-1.2.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jackson-jaxrs-json-provider-2.10.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jackson-jaxrs-base-2.10.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-yarn-client-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/commons-collections4-4.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jetty-server-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jetty-http-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/jetty-io-9.3.24.v20180605.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/commons-daemon-1.0.13.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/leveldbjni-all-1.8.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-hdfs-client-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/okhttp-2.7.5.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/okio-1.6.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/RoaringBitmap-0.4.9.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/async-http-client-1.9.40.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-mapreduce-client-core-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-mapreduce-client-common-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-yarn-server-web-proxy-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-yarn-server-common-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-yarn-registry-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/dnsjava-2.1.7.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/ehcache-3.3.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/HikariCP-java7-2.4.12.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/mssql-jdbc-6.2.1.jre7.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/bcprov-jdk15on-1.60.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/bcpkix-jdk15on-1.60.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/servlet-api-2.5.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/metrics-core-3.1.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-yarn-server-applicationhistoryservice-3.1.1.3.1.5.0-152.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/objenesis-1.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/fst-2.50.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/java-util-1.9.0.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/json-io-2.5.1.jar:/data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tezlib/lib/hadoop-yarn-server-timeline-pluginstorage-3.1.1.3.1.5.0-152.jar: java.io.tmpdir: /data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tmp user.dir: /data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001 user.name: yarn ************************************************************/ 2021-01-29 13:32:06,217 [INFO] [main] |shim.HadoopShimsLoader|: Trying to locate HadoopShimProvider for hadoopVersion=3.1.1.3.1.5.0-152, majorVersion=3, minorVersion=1 2021-01-29 13:32:06,219 [INFO] [main] |shim.HadoopShimsLoader|: Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=3.1.1.3.1.5.0-152, majorVersion=3, minorVersion=1 2021-01-29 13:32:06,245 [INFO] [main] |app.DAGAppMaster|: AM Level configured TaskSchedulers: [0:TezYarn:null],[1:TezUber:null] 2021-01-29 13:32:06,245 [INFO] [main] |app.DAGAppMaster|: AM Level configured ContainerLaunchers: [0:TezYarn:null],[1:TezUber:null] 2021-01-29 13:32:06,245 [INFO] [main] |app.DAGAppMaster|: AM Level configured TaskCommunicators: [0:TezYarn:null],[1:TezUber:null] 2021-01-29 13:32:06,245 [INFO] [main] |app.DAGAppMaster|: Comparing client version with AM version, clientVersion=0.9.1.3.1.5.0-152, AMVersion=0.9.1.3.1.5.0-152 2021-01-29 13:32:07,001 [INFO] [main] |app.TaskCommunicatorManager|: Creating Default Task Communicator 2021-01-29 13:32:07,009 [INFO] [main] |Configuration.deprecation|: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive 2021-01-29 13:32:07,011 [INFO] [main] |Configuration.deprecation|: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 2021-01-29 13:32:07,012 [INFO] [main] |app.TaskCommunicatorManager|: Creating Default Local Task Communicator 2021-01-29 13:32:07,020 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap 2021-01-29 13:32:07,021 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker 2021-01-29 13:32:07,022 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler 2021-01-29 13:32:07,022 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher 2021-01-29 13:32:07,022 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher 2021-01-29 13:32:07,023 [INFO] [main] |app.DAGAppMaster|: Using concurrent dispatcher: false 2021-01-29 13:32:07,023 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher 2021-01-29 13:32:07,023 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher 2021-01-29 13:32:07,024 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler 2021-01-29 13:32:07,024 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler 2021-01-29 13:32:07,042 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager 2021-01-29 13:32:07,045 [INFO] [main] |launcher.TezContainerLauncherImpl|: Creating DefaultContainerLauncher 2021-01-29 13:32:07,052 [INFO] [main] |launcher.TezContainerLauncherImpl|: Upper limit on the thread pool size is 500 2021-01-29 13:32:07,052 [INFO] [main] |launcher.TezContainerLauncherImpl|: Creating LocalContainerLauncher 2021-01-29 13:32:07,062 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager 2021-01-29 13:32:07,096 [INFO] [main] |node.AMNodeTracker|: blacklistDisablePercent is 33, blacklistingEnabled: true, maxTaskFailuresPerNode: 10, nodeUpdatesRescheduleEnabled: false 2021-01-29 13:32:07,097 [INFO] [main] |web.WebUIService|: Tez UI History URL: http://hdp-hive-server.s.2345inc.com:9999/tez-ui/?viewPath=%2F%23%2Ftez-app%2Fapplication_1611750110647_64528 2021-01-29 13:32:07,098 [INFO] [main] |history.HistoryEventHandler|: Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService 2021-01-29 13:32:07,186 [INFO] [main] |proto.ProtoHistoryLoggingService|: Initing ProtoHistoryLoggingService 2021-01-29 13:32:07,186 [INFO] [main] |proto.ProtoHistoryLoggingService|: Inited ProtoHistoryLoggingService 2021-01-29 13:32:07,186 [INFO] [main] |recovery.RecoveryService|: RecoveryService initialized with recoveryPath=hdfs://ztcluster/tmp/hive/data_quality/_tez_session_dir/74300ba4-ffcf-43e7-87c4-a2ea8a280d1c/.tez/application_1611750110647_64528/recovery/1, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100 2021-01-29 13:32:07,187 [INFO] [main] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:N/A][Event:APP_LAUNCHED]: applicationId=application_1611750110647_64528, appSubmitTime=1611898322753, launchTime=1611898326136 2021-01-29 13:32:07,187 [INFO] [main] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1611750110647_64528_000001, appSubmitTime=1611898322753, launchTime=1611898326136 2021-01-29 13:32:07,191 [INFO] [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler] |proto.ProtoHistoryLoggingService|: Starting ProtoHistoryLoggingService 2021-01-29 13:32:07,205 [INFO] [ServiceThread:org.apache.tez.dag.app.TaskCommunicatorManager] |ipc.CallQueueManager|: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 2021-01-29 13:32:07,205 [INFO] [ServiceThread:DAGClientRPCServer] |ipc.CallQueueManager|: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 200 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 2021-01-29 13:32:07,220 [INFO] [Socket Reader #1 for port 17961] |ipc.Server|: Starting Socket Reader #1 for port 17961 2021-01-29 13:32:07,220 [INFO] [Socket Reader #1 for port 27287] |ipc.Server|: Starting Socket Reader #1 for port 27287 2021-01-29 13:32:07,241 [INFO] [IPC Server Responder] |ipc.Server|: IPC Server Responder: starting 2021-01-29 13:32:07,242 [INFO] [IPC Server listener on 17961] |ipc.Server|: IPC Server listener on 17961: starting 2021-01-29 13:32:07,242 [INFO] [IPC Server Responder] |ipc.Server|: IPC Server Responder: starting 2021-01-29 13:32:07,242 [INFO] [IPC Server listener on 27287] |ipc.Server|: IPC Server listener on 27287: starting 2021-01-29 13:32:07,243 [INFO] [ServiceThread:DAGClientRPCServer] |client.DAGClientServer|: Instantiated DAGClientRPCServer at wx12-dsj-node045/172.22.64.145:17961 2021-01-29 13:32:07,245 [INFO] [ServiceThread:org.apache.tez.dag.app.TaskCommunicatorManager] |app.TezTaskCommunicatorImpl|: Instantiated TezTaskCommunicator RPC at wx12-dsj-node045/172.22.64.145:27287 2021-01-29 13:32:07,247 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |util.log|: Logging initialized @2156ms 2021-01-29 13:32:07,305 [INFO] [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler] |zlib.ZlibFactory|: Successfully loaded & initialized native-zlib library 2021-01-29 13:32:07,306 [INFO] [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler] |compress.CodecPool|: Got brand-new compressor [.deflate] 2021-01-29 13:32:07,315 [INFO] [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler] |proto.ProtoHistoryLoggingService|: Started ProtoHistoryLoggingService 2021-01-29 13:32:07,359 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |server.AuthenticationFilter|: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2021-01-29 13:32:07,362 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpRequestLog|: Http request log for http.requests. is not defined 2021-01-29 13:32:07,366 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2021-01-29 13:32:07,396 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context 2021-01-29 13:32:07,396 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static 2021-01-29 13:32:07,398 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: adding path spec: /* 2021-01-29 13:32:07,727 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |webapp.WebApps|: Registered webapp guice modules 2021-01-29 13:32:07,728 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Jetty bound to port 7374 2021-01-29 13:32:07,730 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |server.Server|: jetty-9.3.24.v20180605, build timestamp: 2018-06-06T01:11:56+08:00, git hash: 84205aa28f11a4f31f2a3b86d1bba2cc8ab69827 2021-01-29 13:32:07,767 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |handler.ContextHandler|: Started o.e.j.s.ServletContextHandler@1b6984ef{/static,jar:file:/data01/hdp/yarn/local/filecache/2160/tez.tar.gz/lib/hadoop-yarn-common-3.1.1.3.1.5.0-152.jar!/webapps/static,AVAILABLE} 2021-01-29 13:32:10,271 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |handler.ContextHandler|: Started o.e.j.w.WebAppContext@614cc5f0{/,file:///data01/hdp/yarn/local/usercache/data_quality/appcache/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/tmp/jetty-0.0.0.0-7374-webapps-_-any-132207797389857526.dir/webapp/,AVAILABLE}{/} 2021-01-29 13:32:10,275 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |server.AbstractConnector|: Started ServerConnector@463f922f{HTTP/1.1,[http/1.1]}{0.0.0.0:7374} 2021-01-29 13:32:10,275 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |server.Server|: Started @5186ms 2021-01-29 13:32:10,275 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |webapp.WebApps|: Web app started at 7374 2021-01-29 13:32:10,275 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |web.WebUIService|: Instantiated WebUIService at http://wx12-dsj-node045:7374/ui/ 2021-01-29 13:32:10,278 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.TaskSchedulerManager|: Creating YARN TaskScheduler: org.apache.tez.dag.app.rm.YarnTaskSchedulerService 2021-01-29 13:32:10,307 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |conf.Configuration|: resource-types.xml not found 2021-01-29 13:32:10,307 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |resource.ResourceUtils|: Unable to find 'resource-types.xml'. 2021-01-29 13:32:10,323 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |Configuration.deprecation|: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 2021-01-29 13:32:10,324 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.TaskSchedulerManager|: Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=111101111 2021-01-29 13:32:10,332 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.YarnTaskSchedulerService|: YarnTaskScheduler initialized with configuration: maxRMHeartbeatInterval: 250, containerReuseEnabled: true, reuseRackLocal: true, reuseNonLocal: false, localitySchedulingDelay: 250, preemptionPercentage: 10, preemptionMaxWaitTime: 60000, numHeartbeatsBetweenPreemptions: 3, idleContainerMinTimeout: 10000, idleContainerMaxTimeout: 20000, sessionMinHeldContainers: 0 2021-01-29 13:32:10,373 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |client.ConfiguredRMFailoverProxyProvider|: Failing over to rm2 2021-01-29 13:32:10,424 [INFO] [main] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1611750110647_64528_000001, startTime=1611898330423 2021-01-29 13:32:10,424 [INFO] [main] |app.DAGAppMaster|: In Session mode. Waiting for DAG over RPC 2021-01-29 13:32:10,464 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: App total resource memory: 526336 cpu: 410 taskAllocations: 0 2021-01-29 13:32:10,465 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: Free: pendingRequests: 0 delayedContainers: 0 heartbeats: 1 lastPreemptionHeartbeat: 0 2021-01-29 13:32:10,470 [INFO] [Dispatcher thread {Central}] |node.PerSourceNodeTracker|: Num cluster nodes = 75 2021-01-29 13:32:10,737 [INFO] [IPC Server handler 1 on 17961] |app.DAGAppMaster|: Starting DAG submitted via RPC: select count(1) as counts, 0 a...'11' (Stage-1) 2021-01-29 13:32:10,738 [INFO] [IPC Server handler 1 on 17961] |app.DAGAppMaster|: Merging AM credentials into DAG credentials 2021-01-29 13:32:10,768 [INFO] [IPC Server handler 1 on 17961] |app.DAGAppMaster|: Generating DAG graphviz file, dagId=dag_1611750110647_64528_1, filePath=/data01/hdp/yarn/log/application_1611750110647_64528/container_e703_1611750110647_64528_01_000001/dag_1611750110647_64528_1.dot 2021-01-29 13:32:10,770 [INFO] [IPC Server handler 1 on 17961] |common.TezUtilsInternal|: Redirecting log file based on addend: dag_1611750110647_64528_1 End of LogType:syslog *********************************************************************** Container: container_e703_1611750110647_64528_01_000001 on wx12-dsj-node045_45454_1611898349701 LogAggregationType: AGGREGATED =============================================================================================== LogType:syslog_dag_1611750110647_64528_1 LogLastModifiedTime:Fri Jan 29 13:32:30 +0800 2021 LogLength:44380 LogContents: 2021-01-29 13:32:10,770 [INFO] [IPC Server handler 1 on 17961] |app.DAGAppMaster|: Running DAG: select count(1) as counts, 0 a...'11' (Stage-1), callerContext={ context=HIVE, callerType=HIVE_QUERY_ID, callerId=hive_20210129133157_63a0c95d-f2f9-4c13-b252-de43cbc06abc } 2021-01-29 13:32:12,727 [INFO] [IPC Server handler 1 on 17961] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:DAG_SUBMITTED]: dagID=dag_1611750110647_64528_1, submitTime=1611898330737, queueName=adhoc 2021-01-29 13:32:12,737 [INFO] [HistoryEventHandlingThread] |compress.CodecPool|: Got brand-new compressor [.deflate] 2021-01-29 13:32:12,792 [INFO] [IPC Server handler 1 on 17961] |impl.VertexImpl|: Default container context for vertex_1611750110647_64528_1_00 [Map 1]=LocalResources: [[ name=hive-hcatalog-core.jar, value=resource { scheme: "hdfs" host: "ztcluster" port: -1 file: "/tmp/hive/data_quality/_tez_session_dir/74300ba4-ffcf-43e7-87c4-a2ea8a280d1c-resources/hive-hcatalog-core.jar" } size: 269276 timestamp: 1611898322619 type: FILE visibility: PRIVATE],[ name=tez-conf.pb, value=resource { scheme: "hdfs" host: "ztcluster" port: -1 file: "/tmp/hive/data_quality/_tez_session_dir/74300ba4-ffcf-43e7-87c4-a2ea8a280d1c/.tez/application_1611750110647_64528/tez-conf.pb" } size: 127507 timestamp: 1611898322714 type: FILE visibility: APPLICATION],[ name=hive-exec-3.1.0.3.1.5.0-152-52ac635d1c2eaf5d3e864f9c6dd1cc6c640a4133a03a6747e20ad2156829ebdf.jar, value=resource { scheme: "hdfs" host: "ztcluster" port: -1 file: "/user/data_quality/.hiveJars/hive-exec-3.1.0.3.1.5.0-152-52ac635d1c2eaf5d3e864f9c6dd1cc6c640a4133a03a6747e20ad2156829ebdf.jar" } size: 42785708 timestamp: 1600767139232 type: FILE visibility: PRIVATE],[ name=tezlib, value=resource { scheme: "hdfs" host: "ztcluster" port: -1 file: "/hdp/apps/3.1.5.0-152/tez/tez.tar.gz" } size: 282770011 timestamp: 1590679864288 type: ARCHIVE visibility: PUBLIC],], environment: [[ SHELL=/bin/bash ],[ LD_LIBRARY_PATH=$PWD:/usr/hdp/3.1.5.0-152/hadoop/lib/native:/usr/hdp/3.1.5.0-152/hadoop/lib/native/Linux-amd64-64:$PWD:/usr/hdp/3.1.5.0-152/hadoop/lib/native:/usr/hdp/3.1.5.0-152/hadoop/lib/native/Linux-amd64-64:$PWD:/usr/hdp/3.1.5.0-152/hadoop/lib/native:/usr/hdp/3.1.5.0-152/hadoop/lib/native/Linux-amd64-64:$PWD:/usr/hdp/3.1.5.0-152/hadoop/lib/native:/usr/hdp/3.1.5.0-152/hadoop/lib/native/Linux-amd64-64:$HADOOP_COMMON_HOME/lib/native/ ],[ HADOOP_ROOT_LOGGER=INFO,CLA ],[ CLASSPATH=/usr/hdp/3.1.5.0-152/hadoop/lib/hadoop-lzo-0.6.0.3.1.5.0-152.jar:/etc/hadoop/conf/secure:$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*: ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhdp.version=3.1.5.0-152 -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir= -Dtez.root.logger=INFO,CLA , vertex: vertex_1611750110647_64528_1_00 [Map 1], Default Resources= 2021-01-29 13:32:12,793 [INFO] [IPC Server handler 1 on 17961] |impl.VertexImpl|: Setting 1 additional inputs for vertexvertex_1611750110647_64528_1_00 [Map 1] 2021-01-29 13:32:12,794 [INFO] [IPC Server handler 1 on 17961] |impl.VertexImpl|: Using ExecutionContext from Vertex for Vertex Map 1 2021-01-29 13:32:12,794 [INFO] [IPC Server handler 1 on 17961] |impl.VertexImpl|: Running vertex: vertex_1611750110647_64528_1_00 [Map 1] : TaskScheduler=0:TezYarn, ContainerLauncher=0:TezYarn, TaskCommunicator=0:TezYarn 2021-01-29 13:32:12,797 [INFO] [IPC Server handler 1 on 17961] |impl.VertexImpl|: Default container context for vertex_1611750110647_64528_1_01 [Reducer 2]=LocalResources: [[ name=hive-hcatalog-core.jar, value=resource { scheme: "hdfs" host: "ztcluster" port: -1 file: "/tmp/hive/data_quality/_tez_session_dir/74300ba4-ffcf-43e7-87c4-a2ea8a280d1c-resources/hive-hcatalog-core.jar" } size: 269276 timestamp: 1611898322619 type: FILE visibility: PRIVATE],[ name=tez-conf.pb, value=resource { scheme: "hdfs" host: "ztcluster" port: -1 file: "/tmp/hive/data_quality/_tez_session_dir/74300ba4-ffcf-43e7-87c4-a2ea8a280d1c/.tez/application_1611750110647_64528/tez-conf.pb" } size: 127507 timestamp: 1611898322714 type: FILE visibility: APPLICATION],[ name=hive-exec-3.1.0.3.1.5.0-152-52ac635d1c2eaf5d3e864f9c6dd1cc6c640a4133a03a6747e20ad2156829ebdf.jar, value=resource { scheme: "hdfs" host: "ztcluster" port: -1 file: "/user/data_quality/.hiveJars/hive-exec-3.1.0.3.1.5.0-152-52ac635d1c2eaf5d3e864f9c6dd1cc6c640a4133a03a6747e20ad2156829ebdf.jar" } size: 42785708 timestamp: 1600767139232 type: FILE visibility: PRIVATE],[ name=tezlib, value=resource { scheme: "hdfs" host: "ztcluster" port: -1 file: "/hdp/apps/3.1.5.0-152/tez/tez.tar.gz" } size: 282770011 timestamp: 1590679864288 type: ARCHIVE visibility: PUBLIC],], environment: [[ SHELL=/bin/bash ],[ LD_LIBRARY_PATH=$PWD:/usr/hdp/3.1.5.0-152/hadoop/lib/native:/usr/hdp/3.1.5.0-152/hadoop/lib/native/Linux-amd64-64:$PWD:/usr/hdp/3.1.5.0-152/hadoop/lib/native:/usr/hdp/3.1.5.0-152/hadoop/lib/native/Linux-amd64-64:$PWD:/usr/hdp/3.1.5.0-152/hadoop/lib/native:/usr/hdp/3.1.5.0-152/hadoop/lib/native/Linux-amd64-64:$PWD:/usr/hdp/3.1.5.0-152/hadoop/lib/native:/usr/hdp/3.1.5.0-152/hadoop/lib/native/Linux-amd64-64:$HADOOP_COMMON_HOME/lib/native/ ],[ HADOOP_ROOT_LOGGER=INFO,CLA ],[ CLASSPATH=/usr/hdp/3.1.5.0-152/hadoop/lib/hadoop-lzo-0.6.0.3.1.5.0-152.jar:/etc/hadoop/conf/secure:$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*: ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhdp.version=3.1.5.0-152 -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir= -Dtez.root.logger=INFO,CLA , vertex: vertex_1611750110647_64528_1_01 [Reducer 2], Default Resources= 2021-01-29 13:32:12,797 [INFO] [IPC Server handler 1 on 17961] |impl.VertexImpl|: Setting 1 additional outputs for vertex vertex_1611750110647_64528_1_01 [Reducer 2] 2021-01-29 13:32:12,798 [INFO] [IPC Server handler 1 on 17961] |impl.VertexImpl|: Using ExecutionContext from Vertex for Vertex Reducer 2 2021-01-29 13:32:12,798 [INFO] [IPC Server handler 1 on 17961] |impl.VertexImpl|: Running vertex: vertex_1611750110647_64528_1_01 [Reducer 2] : TaskScheduler=0:TezYarn, ContainerLauncher=0:TezYarn, TaskCommunicator=0:TezYarn 2021-01-29 13:32:12,805 [INFO] [IPC Server handler 1 on 17961] |impl.DAGImpl|: Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrder 2021-01-29 13:32:12,808 [INFO] [IPC Server handler 1 on 17961] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:DAG_INITIALIZED]: dagID=dag_1611750110647_64528_1, initTime=1611898332807 2021-01-29 13:32:12,808 [INFO] [IPC Server handler 1 on 17961] |impl.DAGImpl|: dag_1611750110647_64528_1 transitioned from NEW to INITED due to event DAG_INIT 2021-01-29 13:32:12,810 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Added additional resources : [[]] to classpath 2021-01-29 13:32:12,812 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:DAG_STARTED]: dagID=dag_1611750110647_64528_1, startTime=1611898332810 2021-01-29 13:32:12,813 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: dag_1611750110647_64528_1 transitioned from INITED to RUNNING due to event DAG_START 2021-01-29 13:32:12,813 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Root Inputs exist for Vertex: Map 1 : {dm_common_helper_detect_click_last_trace_code_h={InputName=dm_common_helper_detect_click_last_trace_code_h}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInputLegacy, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator, hasPayload=false}} 2021-01-29 13:32:12,814 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Starting root input initializer for input: dm_common_helper_detect_click_last_trace_code_h, with class: [org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator] 2021-01-29 13:32:12,814 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Setting vertexManager to RootInputVertexManager for vertex_1611750110647_64528_1_00 [Map 1] 2021-01-29 13:32:12,848 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1611750110647_64528_1_00 [Map 1] 2021-01-29 13:32:12,848 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Vertex will initialize from input initializer. vertex_1611750110647_64528_1_00 [Map 1] 2021-01-29 13:32:12,850 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Starting 1 inputInitializers for vertex vertex_1611750110647_64528_1_00 [Map 1] 2021-01-29 13:32:12,865 [INFO] [Dispatcher thread {Central}] |Configuration.deprecation|: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed 2021-01-29 13:32:13,018 [INFO] [Dispatcher thread {Central}] |conf.HiveConf|: Found configuration file null 2021-01-29 13:32:13,021 [INFO] [Dispatcher thread {Central}] |exec.Utilities|: PLAN PATH = hdfs://ztcluster/tmp/hive/data_quality/fe538bf7-9f34-430a-9619-0b0e1b97b402/hive_2021-01-29_13-31-57_274_5600475007733480004-168252/data_quality/_tez_scratch_dir/2ba4f9da-44e5-4688-aece-e3045e3b3c17/map.xml 2021-01-29 13:32:13,029 [INFO] [Dispatcher thread {Central}] |exec.SerializationUtilities|: Deserializing MapWork using kryo 2021-01-29 13:32:13,223 [INFO] [Dispatcher thread {Central}] |exec.Utilities|: Deserialized plan (via RPC) - name: Map 1 size: 4.67KB 2021-01-29 13:32:13,224 [INFO] [Dispatcher thread {Central}] |tez.HiveSplitGenerator|: SplitGenerator using llap affinitized locations: false 2021-01-29 13:32:13,224 [INFO] [Dispatcher thread {Central}] |tez.HiveSplitGenerator|: SplitLocationProvider: org.apache.hadoop.hive.ql.exec.tez.Utils$1@36e3e63b 2021-01-29 13:32:13,233 [INFO] [InputInitializer {Map 1} #0] |dag.RootInputInitializerManager|: Starting InputInitializer for Input: dm_common_helper_detect_click_last_trace_code_h on vertex vertex_1611750110647_64528_1_00 [Map 1] 2021-01-29 13:32:13,235 [INFO] [InputInitializer {Map 1} #0] |tez.HiveSplitGenerator|: GenerateConsistentSplitsInHive=true 2021-01-29 13:32:13,237 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1611750110647_64528_1_00 [Map 1] transitioned from NEW to INITIALIZING due to event V_INIT 2021-01-29 13:32:13,242 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Setting vertexManager to ShuffleVertexManager for vertex_1611750110647_64528_1_01 [Reducer 2] 2021-01-29 13:32:13,247 [INFO] [InputInitializer {Map 1} #0] |tez.HiveSplitGenerator|: The preferred split size is 16777216 2021-01-29 13:32:13,247 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: PLAN PATH = hdfs://ztcluster/tmp/hive/data_quality/fe538bf7-9f34-430a-9619-0b0e1b97b402/hive_2021-01-29_13-31-57_274_5600475007733480004-168252/data_quality/_tez_scratch_dir/2ba4f9da-44e5-4688-aece-e3045e3b3c17/map.xml 2021-01-29 13:32:13,250 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: Processing alias dm_common_helper_detect_click_last_trace_code_h 2021-01-29 13:32:13,251 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: Adding 1 inputs; the first input is hdfs://ztcluster/user/hive/warehouse/dm_db.db/dm_common_helper_detect_click_last_trace_code_h/p_dt=2021-01-29/p_hours=11 2021-01-29 13:32:13,257 [INFO] [InputInitializer {Map 1} #0] |io.HiveInputFormat|: hive.io.file.readcolumn.ids = 2021-01-29 13:32:13,257 [INFO] [InputInitializer {Map 1} #0] |io.HiveInputFormat|: hive.io.file.readcolumn.names = 2021-01-29 13:32:13,257 [INFO] [InputInitializer {Map 1} #0] |io.HiveInputFormat|: Generating splits for dirs: hdfs://ztcluster/user/hive/warehouse/dm_db.db/dm_common_helper_detect_click_last_trace_code_h/p_dt=2021-01-29/p_hours=11 2021-01-29 13:32:13,265 [INFO] [Dispatcher thread {Central}] |vertexmanager.ShuffleVertexManagerBase|: Settings minFrac: 0.2 maxFrac: 0.4 auto: false desiredTaskIput: 104857600 2021-01-29 13:32:13,265 [INFO] [Dispatcher thread {Central}] |vertexmanager.ShuffleVertexManager|: minTaskParallelism 1 2021-01-29 13:32:13,266 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Creating 1 tasks for vertex: vertex_1611750110647_64528_1_01 [Reducer 2] 2021-01-29 13:32:13,274 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Directly initializing vertex: vertex_1611750110647_64528_1_01 [Reducer 2] 2021-01-29 13:32:13,276 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1611750110647_64528_1_01, reconfigureDoneTime=1611898333275, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false 2021-01-29 13:32:13,276 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Setting up committers for vertex vertex_1611750110647_64528_1_01 [Reducer 2], numAdditionalOutputs=1 2021-01-29 13:32:13,277 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:VERTEX_INITIALIZED]: vertexName=Reducer 2, vertexId=vertex_1611750110647_64528_1_01, initRequestedTime=1611898333242, initedTime=1611898333276, numTasks=1, processorName=org.apache.hadoop.hive.ql.exec.tez.ReduceTezProcessor, additionalInputsCount=0, initGeneratedEventsCount=0, servicePluginInfo=ServicePluginInfo {containerLauncherName=TezYarn, taskSchedulerName=TezYarn, taskCommunicatorName=TezYarn, containerLauncherClassName=org.apache.tez.dag.app.launcher.TezContainerLauncherImpl, taskSchedulerClassName=org.apache.tez.dag.app.rm.YarnTaskSchedulerService, taskCommunicatorClassName=org.apache.tez.dag.app.TezTaskCommunicatorImpl } 2021-01-29 13:32:13,277 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1611750110647_64528_1_01 [Reducer 2] transitioned from NEW to INITED due to event V_INIT 2021-01-29 13:32:13,303 [INFO] [InputInitializer {Map 1} #0] |orc.OrcInputFormat|: ORC pushdown predicate: leaf-0 = (EQUALS p_dt 2021-01-29), leaf-1 = (EQUALS p_hours 11), expr = (and leaf-0 leaf-1) 2021-01-29 13:32:13,304 [INFO] [InputInitializer {Map 1} #0] |Configuration.deprecation|: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 2021-01-29 13:32:13,332 [INFO] [ORC_GET_SPLITS #1] |exec.Utilities|: PLAN PATH = hdfs://ztcluster/tmp/hive/data_quality/fe538bf7-9f34-430a-9619-0b0e1b97b402/hive_2021-01-29_13-31-57_274_5600475007733480004-168252/data_quality/_tez_scratch_dir/2ba4f9da-44e5-4688-aece-e3045e3b3c17/map.xml 2021-01-29 13:32:13,437 [INFO] [ORC_GET_SPLITS #1] |impl.OrcCodecPool|: Got brand-new codec ZLIB 2021-01-29 13:32:13,481 [INFO] [InputInitializer {Map 1} #0] |orc.OrcInputFormat|: FooterCacheHitRatio: 0/1 2021-01-29 13:32:13,482 [INFO] [InputInitializer {Map 1} #0] |io.HiveInputFormat|: number of splits 1 2021-01-29 13:32:13,483 [INFO] [InputInitializer {Map 1} #0] |tez.HiveSplitGenerator|: Number of input splits: 1. 73 available slots, 1.7 waves. Input format is: org.apache.hadoop.hive.ql.io.HiveInputFormat 2021-01-29 13:32:13,484 [INFO] [InputInitializer {Map 1} #0] |counters.Limits|: Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=3000, COUNTER_NAME_MAX=64, MAX_COUNTERS=10000 2021-01-29 13:32:13,489 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: Processing alias dm_common_helper_detect_click_last_trace_code_h 2021-01-29 13:32:13,490 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: Adding 1 inputs; the first input is hdfs://ztcluster/user/hive/warehouse/dm_db.db/dm_common_helper_detect_click_last_trace_code_h/p_dt=2021-01-29/p_hours=11 2021-01-29 13:32:13,490 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: PLAN PATH = hdfs://ztcluster/tmp/hive/data_quality/fe538bf7-9f34-430a-9619-0b0e1b97b402/hive_2021-01-29_13-31-57_274_5600475007733480004-168252/data_quality/_tez_scratch_dir/2ba4f9da-44e5-4688-aece-e3045e3b3c17/map.xml 2021-01-29 13:32:13,492 [INFO] [InputInitializer {Map 1} #0] |exec.SerializationUtilities|: Deserializing MapWork using kryo 2021-01-29 13:32:13,502 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: Deserialized plan (via RPC) - name: Map 1 size: 4.67KB 2021-01-29 13:32:13,506 [INFO] [InputInitializer {Map 1} #0] |tez.SplitGrouper|: # Src groups for split generation: 2 2021-01-29 13:32:13,509 [INFO] [InputInitializer {Map 1} #0] |tez.SplitGrouper|: Estimated number of tasks: 124 for bucket 1 2021-01-29 13:32:13,512 [INFO] [InputInitializer {Map 1} #0] |grouper.TezSplitGrouper|: Grouping splits in Tez 2021-01-29 13:32:13,513 [INFO] [InputInitializer {Map 1} #0] |grouper.TezSplitGrouper|: Desired splits: 1 too large. Desired splitLength: 7 Min splitLength: 16777216 New desired splits: 1 Final desired splits: 1 All splits have localhost: false Total length: 876 Original splits: 1 2021-01-29 13:32:13,513 [INFO] [InputInitializer {Map 1} #0] |grouper.TezSplitGrouper|: Using original number of splits: 1 desired splits: 1 2021-01-29 13:32:13,515 [INFO] [InputInitializer {Map 1} #0] |tez.SplitGrouper|: Original split count is 1 grouped split count is 1, for bucket: 1 2021-01-29 13:32:13,516 [INFO] [InputInitializer {Map 1} #0] |tez.HiveSplitGenerator|: Number of split groups: 1 2021-01-29 13:32:13,524 [INFO] [InputInitializer {Map 1} #0] |dag.RootInputInitializerManager|: Succeeded InputInitializer for Input: dm_common_helper_detect_click_last_trace_code_h on vertex vertex_1611750110647_64528_1_00 [Map 1] 2021-01-29 13:32:13,529 [INFO] [App Shared Pool - #0] |impl.VertexImpl|: Cannot init vertex: vertex_1611750110647_64528_1_00 [Map 1] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true 2021-01-29 13:32:13,529 [INFO] [App Shared Pool - #0] |impl.VertexImpl|: Got updated RootInputsSpecs: {dm_common_helper_detect_click_last_trace_code_h=forAllWorkUnits=true, update=[1]} 2021-01-29 13:32:13,530 [INFO] [App Shared Pool - #0] |impl.VertexImpl|: Vertex vertex_1611750110647_64528_1_00 [Map 1] parallelism set to 1 2021-01-29 13:32:13,530 [INFO] [App Shared Pool - #0] |impl.VertexImpl|: Cannot init vertex: vertex_1611750110647_64528_1_00 [Map 1] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true 2021-01-29 13:32:13,552 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:VERTEX_INITIALIZED]: vertexName=Map 1, vertexId=vertex_1611750110647_64528_1_00, initRequestedTime=1611898332813, initedTime=1611898333552, numTasks=1, processorName=org.apache.hadoop.hive.ql.exec.tez.MapTezProcessor, additionalInputsCount=1, initGeneratedEventsCount=1, servicePluginInfo=ServicePluginInfo {containerLauncherName=TezYarn, taskSchedulerName=TezYarn, taskCommunicatorName=TezYarn, containerLauncherClassName=org.apache.tez.dag.app.launcher.TezContainerLauncherImpl, taskSchedulerClassName=org.apache.tez.dag.app.rm.YarnTaskSchedulerService, taskCommunicatorClassName=org.apache.tez.dag.app.TezTaskCommunicatorImpl } 2021-01-29 13:32:13,552 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Triggering start event for vertex: vertex_1611750110647_64528_1_00 [Map 1] with distanceFromRoot: 0 2021-01-29 13:32:13,553 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1611750110647_64528_1_00 [Map 1] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION 2021-01-29 13:32:13,554 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:VERTEX_STARTED]: vertexId=vertex_1611750110647_64528_1_00, startRequestedTime=1611898333241, startedTime=1611898333553 2021-01-29 13:32:13,555 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1611750110647_64528_1_00, reconfigureDoneTime=1611898333554, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@9098213f, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true 2021-01-29 13:32:13,555 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1611750110647_64528_1_00 [Map 1] transitioned from INITED to RUNNING due to event V_START 2021-01-29 13:32:13,556 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Source vertex started: vertex_1611750110647_64528_1_00 for vertex: vertex_1611750110647_64528_1_01 [Reducer 2] numStartedSources: 1 numSources: 1 2021-01-29 13:32:13,556 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Triggering start event for vertex: vertex_1611750110647_64528_1_01 [Reducer 2] with distanceFromRoot: 1 2021-01-29 13:32:13,556 [INFO] [App Shared Pool - #0] |impl.RootInputVertexManager|: All source tasks completed. Ramping up 1 remaining tasks for vertex: Map 1 2021-01-29 13:32:13,557 [INFO] [App Shared Pool - #0] |impl.RootInputVertexManager|: Scheduling 1 tasks for vertex: Map 1 with totalTasks: 1. 0 source tasks completed out of 0. MinSourceTaskCompletedFraction: 1.0 min: 0.0 max: 0.0 2021-01-29 13:32:13,557 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:VERTEX_STARTED]: vertexId=vertex_1611750110647_64528_1_01, startRequestedTime=1611898333556, startedTime=1611898333556 2021-01-29 13:32:13,557 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1611750110647_64528_1_01 [Reducer 2] transitioned from INITED to RUNNING due to event V_START 2021-01-29 13:32:13,558 [INFO] [App Shared Pool - #0] |impl.VertexImpl|: Routing pending task events for vertex: vertex_1611750110647_64528_1_00 [Map 1] 2021-01-29 13:32:13,563 [INFO] [App Shared Pool - #1] |vertexmanager.ShuffleVertexManagerBase|: OnVertexStarted vertex: Reducer 2 with 0 source tasks and 1 pending tasks 2021-01-29 13:32:13,564 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:TASK_STARTED]: vertexName=Map 1, taskId=task_1611750110647_64528_1_00_000000, scheduledTime=1611898333562, launchTime=1611898333562 2021-01-29 13:32:13,566 [INFO] [App Shared Pool - #0] |vertexmanager.ShuffleVertexManagerBase|: Received configured notification : CONFIGURED for vertex: Map 1 in vertex: Reducer 2 numBipartiteSourceTasks: 1 2021-01-29 13:32:13,714 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1611750110647_64528_1_00_000000_0 with request: Capability[]Priority[5]AllocationRequestId[0]ExecutionTypeRequest[{Execution Type: GUARANTEED, Enforce Execution Type: false}]Resource Profile[null] host: wx12-dsj-node060 rack: null 2021-01-29 13:32:14,290 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_e703_1611750110647_64528_01_000002, task=attempt_1611750110647_64528_1_00_000000_0, containerHost=wx12-dsj-node008:45454, containerPriority= 5, containerResources=, localityMatchType=RackLocal, matchedLocation=/default-rack, honorLocalityFlags=false, reusedContainer=false, delayedContainers=0 2021-01-29 13:32:14,411 [INFO] [TaskSchedulerAppCallbackExecutor #0] |node.PerSourceNodeTracker|: Adding new node wx12-dsj-node008:45454 to nodeTracker 0 2021-01-29 13:32:14,557 [INFO] [ContainerLauncher #0] |launcher.TezContainerLauncherImpl|: Launching container_e703_1611750110647_64528_01_000002 2021-01-29 13:32:14,765 [INFO] [ContainerLauncher #0] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_e703_1611750110647_64528_01_000002, launchTime=1611898334765 2021-01-29 13:32:18,066 [INFO] [Socket Reader #1 for port 27287] |ipc.Server|: Auth successful for application_1611750110647_64528 (auth:SIMPLE) 2021-01-29 13:32:18,135 [INFO] [Dispatcher thread {Central}] |impl.TaskAttemptImpl|: TaskAttempt: [attempt_1611750110647_64528_1_00_000000_0] submitted. Is using containerId: [container_e703_1611750110647_64528_01_000002] on NM: [wx12-dsj-node008:45454] 2021-01-29 13:32:18,136 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:TASK_ATTEMPT_STARTED]: vertexName=Map 1, taskAttemptId=attempt_1611750110647_64528_1_00_000000_0, startTime=1611898338134, containerId=container_e703_1611750110647_64528_01_000002, nodeId=wx12-dsj-node008:45454 2021-01-29 13:32:18,238 [INFO] [IPC Server handler 1 on 27287] |impl.TaskImpl|: TaskAttempt:attempt_1611750110647_64528_1_00_000000_0 sent events: (0-1). 2021-01-29 13:32:19,892 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=Map 1, taskAttemptId=attempt_1611750110647_64528_1_00_000000_0, creationTime=1611898333593, allocationTime=1611898334520, startTime=1611898338134, finishTime=1611898339889, timeTaken=1755, status=SUCCEEDED, diagnostics= 2021-01-29 13:32:19,908 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Task succeeded with attempt attempt_1611750110647_64528_1_00_000000_0 2021-01-29 13:32:19,916 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:TASK_FINISHED]: vertexName=Map 1, taskId=task_1611750110647_64528_1_00_000000, startTime=1611898338134, finishTime=1611898339911, timeTaken=1777, status=SUCCEEDED, successfulAttemptID=attempt_1611750110647_64528_1_00_000000_0, diagnostics=, counters=Counters: 43, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_WRITTEN=57, HDFS_BYTES_READ=1034, HDFS_READ_OPS=1, HDFS_OP_OPEN=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=0, GC_TIME_MILLIS=66, TASK_DURATION_MILLIS=1755, CPU_MILLISECONDS=6270, PHYSICAL_MEMORY_BYTES=2147483648, VIRTUAL_MEMORY_BYTES=8360558592, COMMITTED_HEAP_BYTES=2147483648, INPUT_RECORDS_PROCESSED=2, INPUT_SPLIT_LENGTH_BYTES=526, OUTPUT_RECORDS=1, OUTPUT_LARGE_RECORDS=0, OUTPUT_BYTES=3, OUTPUT_BYTES_WITH_OVERHEAD=11, OUTPUT_BYTES_PHYSICAL=49, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, DATA_BYTES_VIA_EVENT=0, HIVE, DESERIALIZE_ERRORS=0, RECORDS_IN_Map_1=10, RECORDS_OUT_INTERMEDIATE_Map_1=1, RECORDS_OUT_OPERATOR_GBY_10=1, RECORDS_OUT_OPERATOR_MAP_0=0, RECORDS_OUT_OPERATOR_RS_11=1, RECORDS_OUT_OPERATOR_SEL_9=10, RECORDS_OUT_OPERATOR_TS_0=10, TaskCounter_Map_1_INPUT_dm_common_helper_detect_click_last_trace, INPUT_RECORDS_PROCESSED=2, INPUT_SPLIT_LENGTH_BYTES=526, TaskCounter_Map_1_OUTPUT_Reducer_2, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILL_COUNT=0, DATA_BYTES_VIA_EVENT=0, OUTPUT_BYTES=3, OUTPUT_BYTES_PHYSICAL=49, OUTPUT_BYTES_WITH_OVERHEAD=11, OUTPUT_LARGE_RECORDS=0, OUTPUT_RECORDS=1, SPILLED_RECORDS=0 2021-01-29 13:32:19,923 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Task Completion: vertex_1611750110647_64528_1_00 [Map 1], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null 2021-01-29 13:32:19,923 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: All tasks have succeeded, vertex:vertex_1611750110647_64528_1_00 [Map 1] 2021-01-29 13:32:19,942 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:VERTEX_FINISHED]: vertexName=Map 1, vertexId=vertex_1611750110647_64528_1_00, initRequestedTime=1611898332813, initedTime=1611898333552, startRequestedTime=1611898333241, startedTime=1611898333553, finishTime=1611898339924, timeTaken=6371, status=SUCCEEDED, diagnostics=, counters=Counters: 47, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_WRITTEN=57, HDFS_BYTES_READ=1034, HDFS_READ_OPS=1, HDFS_OP_OPEN=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=0, GC_TIME_MILLIS=66, TASK_DURATION_MILLIS=1755, CPU_MILLISECONDS=6270, PHYSICAL_MEMORY_BYTES=2147483648, VIRTUAL_MEMORY_BYTES=8360558592, COMMITTED_HEAP_BYTES=2147483648, INPUT_RECORDS_PROCESSED=2, INPUT_SPLIT_LENGTH_BYTES=526, OUTPUT_RECORDS=1, OUTPUT_LARGE_RECORDS=0, OUTPUT_BYTES=3, OUTPUT_BYTES_WITH_OVERHEAD=11, OUTPUT_BYTES_PHYSICAL=49, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, DATA_BYTES_VIA_EVENT=0, HIVE, DESERIALIZE_ERRORS=0, RECORDS_IN_Map_1=10, RECORDS_OUT_INTERMEDIATE_Map_1=1, RECORDS_OUT_OPERATOR_GBY_10=1, RECORDS_OUT_OPERATOR_MAP_0=0, RECORDS_OUT_OPERATOR_RS_11=1, RECORDS_OUT_OPERATOR_SEL_9=10, RECORDS_OUT_OPERATOR_TS_0=10, TaskCounter_Map_1_INPUT_dm_common_helper_detect_click_last_trace, INPUT_RECORDS_PROCESSED=2, INPUT_SPLIT_LENGTH_BYTES=526, TaskCounter_Map_1_OUTPUT_Reducer_2, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILL_COUNT=0, DATA_BYTES_VIA_EVENT=0, OUTPUT_BYTES=3, OUTPUT_BYTES_PHYSICAL=49, OUTPUT_BYTES_WITH_OVERHEAD=11, OUTPUT_LARGE_RECORDS=0, OUTPUT_RECORDS=1, SPILLED_RECORDS=0, org.apache.hadoop.hive.ql.exec.tez.HiveInputCounters, GROUPED_INPUT_SPLITS_Map_1=1, INPUT_DIRECTORIES_Map_1=1, INPUT_FILES_Map_1=1, RAW_INPUT_SPLITS_Map_1=1, vertexStats=firstTaskStartTime=1611898338134, firstTasksToStart=[ task_1611750110647_64528_1_00_000000 ], lastTaskFinishTime=1611898339889, lastTasksToFinish=[ task_1611750110647_64528_1_00_000000 ], minTaskDuration=1755, maxTaskDuration=1755, avgTaskDuration=1755.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1611750110647_64528_1_00_000000 ], longestDurationTasks=[ task_1611750110647_64528_1_00_000000 ], vertexTaskStats={numFailedTaskAttempts=0, numKilledTaskAttempts=0, numCompletedTasks=1, numSucceededTasks=1, numKilledTasks=0, numFailedTasks=0}, servicePluginInfo=ServicePluginInfo {containerLauncherName=TezYarn, taskSchedulerName=TezYarn, taskCommunicatorName=TezYarn, containerLauncherClassName=org.apache.tez.dag.app.launcher.TezContainerLauncherImpl, taskSchedulerClassName=org.apache.tez.dag.app.rm.YarnTaskSchedulerService, taskCommunicatorClassName=org.apache.tez.dag.app.TezTaskCommunicatorImpl } 2021-01-29 13:32:19,943 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1611750110647_64528_1_00 [Map 1] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED 2021-01-29 13:32:19,943 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Source task attempt completed for vertex: vertex_1611750110647_64528_1_01 [Reducer 2] attempt: attempt_1611750110647_64528_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING 2021-01-29 13:32:19,944 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Vertex vertex_1611750110647_64528_1_00 [Map 1] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2 2021-01-29 13:32:19,944 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null 2021-01-29 13:32:19,953 [INFO] [App Shared Pool - #1] |vertexmanager.ShuffleVertexManagerBase|: All source tasks completed. Ramping up 1 remaining tasks for vertex: Reducer 2 2021-01-29 13:32:19,953 [INFO] [App Shared Pool - #1] |vertexmanager.ShuffleVertexManagerBase|: Scheduling 1 tasks for vertex: Reducer 2 with totalTasks: 1. 1 source tasks completed out of 1. MinSourceTaskCompletedFraction: 1.0 min: 0.2 max: 0.4 2021-01-29 13:32:19,954 [INFO] [App Shared Pool - #1] |impl.Edge|: Routing to begin for edge: EdgeInfo: sourceVertexName=Map 1, destinationVertexName=Reducer 2. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.UnorderedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.UnorderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true 2021-01-29 13:32:19,954 [INFO] [App Shared Pool - #1] |impl.VertexImpl|: Routing pending task events for vertex: vertex_1611750110647_64528_1_01 [Reducer 2] 2021-01-29 13:32:19,955 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:TASK_STARTED]: vertexName=Reducer 2, taskId=task_1611750110647_64528_1_01_000000, scheduledTime=1611898339955, launchTime=1611898339955 2021-01-29 13:32:19,956 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1611750110647_64528_1_01_000000_0 with request: Capability[]Priority[14]AllocationRequestId[0]ExecutionTypeRequest[{Execution Type: GUARANTEED, Enforce Execution Type: false}]Resource Profile[null] host: null rack: null 2021-01-29 13:32:19,960 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_e703_1611750110647_64528_01_000002, task=attempt_1611750110647_64528_1_01_000000_0, containerHost=wx12-dsj-node008:45454, containerPriority= 5, containerResources=, localityMatchType=NodeLocal, matchedLocation=wx12-dsj-node008, honorLocalityFlags=true, reusedContainer=true, delayedContainers=1 2021-01-29 13:32:19,971 [INFO] [Dispatcher thread {Central}] |impl.TaskAttemptImpl|: TaskAttempt: [attempt_1611750110647_64528_1_01_000000_0] submitted. Is using containerId: [container_e703_1611750110647_64528_01_000002] on NM: [wx12-dsj-node008:45454] 2021-01-29 13:32:19,971 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:TASK_ATTEMPT_STARTED]: vertexName=Reducer 2, taskAttemptId=attempt_1611750110647_64528_1_01_000000_0, startTime=1611898339971, containerId=container_e703_1611750110647_64528_01_000002, nodeId=wx12-dsj-node008:45454 2021-01-29 13:32:20,002 [INFO] [IPC Server handler 9 on 27287] |impl.VertexImpl|: Sending attempt_1611750110647_64528_1_01_000000_0 1 events [0,1) total 1 vertex_1611750110647_64528_1_01 [Reducer 2] 2021-01-29 13:32:20,536 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=Reducer 2, taskAttemptId=attempt_1611750110647_64528_1_01_000000_0, creationTime=1611898339955, allocationTime=1611898339961, startTime=1611898339971, finishTime=1611898340535, timeTaken=564, status=SUCCEEDED, diagnostics= 2021-01-29 13:32:20,536 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Task succeeded with attempt attempt_1611750110647_64528_1_01_000000_0 2021-01-29 13:32:20,539 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:TASK_FINISHED]: vertexName=Reducer 2, taskId=task_1611750110647_64528_1_01_000000, startTime=1611898339971, finishTime=1611898340536, timeTaken=565, status=SUCCEEDED, successfulAttemptID=attempt_1611750110647_64528_1_01_000000_0, diagnostics=, counters=Counters: 43, File System Counters, FILE_BYTES_READ=57, HDFS_BYTES_WRITTEN=106, HDFS_READ_OPS=3, HDFS_WRITE_OPS=2, HDFS_OP_CREATE=1, HDFS_OP_GET_FILE_STATUS=3, HDFS_OP_RENAME=1, org.apache.tez.common.counters.TaskCounter, NUM_SHUFFLED_INPUTS=1, NUM_FAILED_SHUFFLE_INPUTS=0, GC_TIME_MILLIS=66, TASK_DURATION_MILLIS=564, CPU_MILLISECONDS=2630, PHYSICAL_MEMORY_BYTES=2147483648, VIRTUAL_MEMORY_BYTES=8365252608, COMMITTED_HEAP_BYTES=2147483648, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=0, SHUFFLE_BYTES=25, SHUFFLE_BYTES_DECOMPRESSED=11, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=25, SHUFFLE_PHASE_TIME=123, FIRST_EVENT_RECEIVED=102, LAST_EVENT_RECEIVED=102, HIVE, CREATED_FILES=1, RECORDS_OUT_0=1, RECORDS_OUT_INTERMEDIATE_Reducer_2=0, RECORDS_OUT_OPERATOR_FS_14=1, RECORDS_OUT_OPERATOR_GBY_12=1, RECORDS_OUT_OPERATOR_SEL_13=1, TaskCounter_Reducer_2_INPUT_Map_1, FIRST_EVENT_RECEIVED=102, INPUT_RECORDS_PROCESSED=1, LAST_EVENT_RECEIVED=102, NUM_FAILED_SHUFFLE_INPUTS=0, NUM_SHUFFLED_INPUTS=1, SHUFFLE_BYTES=25, SHUFFLE_BYTES_DECOMPRESSED=11, SHUFFLE_BYTES_DISK_DIRECT=25, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_PHASE_TIME=123, TaskCounter_Reducer_2_OUTPUT_out_Reducer_2, OUTPUT_RECORDS=0 2021-01-29 13:32:20,540 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Task Completion: vertex_1611750110647_64528_1_01 [Reducer 2], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null 2021-01-29 13:32:20,540 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: All tasks have succeeded, vertex:vertex_1611750110647_64528_1_01 [Reducer 2] 2021-01-29 13:32:20,544 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:VERTEX_FINISHED]: vertexName=Reducer 2, vertexId=vertex_1611750110647_64528_1_01, initRequestedTime=1611898333242, initedTime=1611898333276, startRequestedTime=1611898333556, startedTime=1611898333556, finishTime=1611898340541, timeTaken=6985, status=SUCCEEDED, diagnostics=, counters=Counters: 43, File System Counters, FILE_BYTES_READ=57, HDFS_BYTES_WRITTEN=106, HDFS_READ_OPS=3, HDFS_WRITE_OPS=2, HDFS_OP_CREATE=1, HDFS_OP_GET_FILE_STATUS=3, HDFS_OP_RENAME=1, org.apache.tez.common.counters.TaskCounter, NUM_SHUFFLED_INPUTS=1, NUM_FAILED_SHUFFLE_INPUTS=0, GC_TIME_MILLIS=66, TASK_DURATION_MILLIS=564, CPU_MILLISECONDS=2630, PHYSICAL_MEMORY_BYTES=2147483648, VIRTUAL_MEMORY_BYTES=8365252608, COMMITTED_HEAP_BYTES=2147483648, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=0, SHUFFLE_BYTES=25, SHUFFLE_BYTES_DECOMPRESSED=11, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=25, SHUFFLE_PHASE_TIME=123, FIRST_EVENT_RECEIVED=102, LAST_EVENT_RECEIVED=102, HIVE, CREATED_FILES=1, RECORDS_OUT_0=1, RECORDS_OUT_INTERMEDIATE_Reducer_2=0, RECORDS_OUT_OPERATOR_FS_14=1, RECORDS_OUT_OPERATOR_GBY_12=1, RECORDS_OUT_OPERATOR_SEL_13=1, TaskCounter_Reducer_2_INPUT_Map_1, FIRST_EVENT_RECEIVED=102, INPUT_RECORDS_PROCESSED=1, LAST_EVENT_RECEIVED=102, NUM_FAILED_SHUFFLE_INPUTS=0, NUM_SHUFFLED_INPUTS=1, SHUFFLE_BYTES=25, SHUFFLE_BYTES_DECOMPRESSED=11, SHUFFLE_BYTES_DISK_DIRECT=25, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_PHASE_TIME=123, TaskCounter_Reducer_2_OUTPUT_out_Reducer_2, OUTPUT_RECORDS=0, vertexStats=firstTaskStartTime=1611898339971, firstTasksToStart=[ task_1611750110647_64528_1_01_000000 ], lastTaskFinishTime=1611898340535, lastTasksToFinish=[ task_1611750110647_64528_1_01_000000 ], minTaskDuration=564, maxTaskDuration=564, avgTaskDuration=564.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1611750110647_64528_1_01_000000 ], longestDurationTasks=[ task_1611750110647_64528_1_01_000000 ], vertexTaskStats={numFailedTaskAttempts=0, numKilledTaskAttempts=0, numCompletedTasks=1, numSucceededTasks=1, numKilledTasks=0, numFailedTasks=0}, servicePluginInfo=ServicePluginInfo {containerLauncherName=TezYarn, taskSchedulerName=TezYarn, taskCommunicatorName=TezYarn, containerLauncherClassName=org.apache.tez.dag.app.launcher.TezContainerLauncherImpl, taskSchedulerClassName=org.apache.tez.dag.app.rm.YarnTaskSchedulerService, taskCommunicatorClassName=org.apache.tez.dag.app.TezTaskCommunicatorImpl } 2021-01-29 13:32:20,544 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1611750110647_64528_1_01 [Reducer 2] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED 2021-01-29 13:32:20,544 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Vertex vertex_1611750110647_64528_1_01 [Reducer 2] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2 2021-01-29 13:32:20,544 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null 2021-01-29 13:32:20,544 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: No exclusive output committers for vertex: vertex_1611750110647_64528_1_01 [Reducer 2] 2021-01-29 13:32:20,544 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: No output committers for vertex: vertex_1611750110647_64528_1_00 [Map 1] 2021-01-29 13:32:20,555 [INFO] [Dispatcher thread {Central}] |recovery.RecoveryService|: DAG completed, dagId=dag_1611750110647_64528_1, queueSize=0 2021-01-29 13:32:20,572 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:DAG_FINISHED]: dagId=dag_1611750110647_64528_1, startTime=1611898332810, finishTime=1611898340544, timeTaken=7734, status=SUCCEEDED, diagnostics=, counters=Counters: 85, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=5020, AM_GC_TIME_MILLIS=0, File System Counters, FILE_BYTES_READ=57, FILE_BYTES_WRITTEN=57, HDFS_BYTES_READ=1034, HDFS_BYTES_WRITTEN=106, HDFS_READ_OPS=4, HDFS_WRITE_OPS=2, HDFS_OP_CREATE=1, HDFS_OP_GET_FILE_STATUS=3, HDFS_OP_OPEN=1, HDFS_OP_RENAME=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_FAILED_SHUFFLE_INPUTS=0, GC_TIME_MILLIS=132, TASK_DURATION_MILLIS=2319, CPU_MILLISECONDS=8900, PHYSICAL_MEMORY_BYTES=4294967296, VIRTUAL_MEMORY_BYTES=16725811200, COMMITTED_HEAP_BYTES=4294967296, INPUT_RECORDS_PROCESSED=3, INPUT_SPLIT_LENGTH_BYTES=526, OUTPUT_RECORDS=1, OUTPUT_LARGE_RECORDS=0, OUTPUT_BYTES=3, OUTPUT_BYTES_WITH_OVERHEAD=11, OUTPUT_BYTES_PHYSICAL=49, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=25, SHUFFLE_BYTES_DECOMPRESSED=11, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=25, SHUFFLE_PHASE_TIME=123, FIRST_EVENT_RECEIVED=102, LAST_EVENT_RECEIVED=102, DATA_BYTES_VIA_EVENT=0, HIVE, CREATED_FILES=1, DESERIALIZE_ERRORS=0, RECORDS_IN_Map_1=10, RECORDS_OUT_0=1, RECORDS_OUT_INTERMEDIATE_Map_1=1, RECORDS_OUT_INTERMEDIATE_Reducer_2=0, RECORDS_OUT_OPERATOR_FS_14=1, RECORDS_OUT_OPERATOR_GBY_10=1, RECORDS_OUT_OPERATOR_GBY_12=1, RECORDS_OUT_OPERATOR_MAP_0=0, RECORDS_OUT_OPERATOR_RS_11=1, RECORDS_OUT_OPERATOR_SEL_13=1, RECORDS_OUT_OPERATOR_SEL_9=10, RECORDS_OUT_OPERATOR_TS_0=10, TaskCounter_Map_1_INPUT_dm_common_helper_detect_click_last_trace, INPUT_RECORDS_PROCESSED=2, INPUT_SPLIT_LENGTH_BYTES=526, TaskCounter_Map_1_OUTPUT_Reducer_2, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILL_COUNT=0, DATA_BYTES_VIA_EVENT=0, OUTPUT_BYTES=3, OUTPUT_BYTES_PHYSICAL=49, OUTPUT_BYTES_WITH_OVERHEAD=11, OUTPUT_LARGE_RECORDS=0, OUTPUT_RECORDS=1, SPILLED_RECORDS=0, TaskCounter_Reducer_2_INPUT_Map_1, FIRST_EVENT_RECEIVED=102, INPUT_RECORDS_PROCESSED=1, LAST_EVENT_RECEIVED=102, NUM_FAILED_SHUFFLE_INPUTS=0, NUM_SHUFFLED_INPUTS=1, SHUFFLE_BYTES=25, SHUFFLE_BYTES_DECOMPRESSED=11, SHUFFLE_BYTES_DISK_DIRECT=25, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_PHASE_TIME=123, TaskCounter_Reducer_2_OUTPUT_out_Reducer_2, OUTPUT_RECORDS=0, org.apache.hadoop.hive.ql.exec.tez.HiveInputCounters, GROUPED_INPUT_SPLITS_Map_1=1, INPUT_DIRECTORIES_Map_1=1, INPUT_FILES_Map_1=1, RAW_INPUT_SPLITS_Map_1=1 2021-01-29 13:32:20,573 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: DAG: dag_1611750110647_64528_1 finished with state: SUCCEEDED 2021-01-29 13:32:20,573 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: dag_1611750110647_64528_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED 2021-01-29 13:32:20,573 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: DAG completed, dagId=dag_1611750110647_64528_1, dagState=SUCCEEDED 2021-01-29 13:32:20,573 [INFO] [Dispatcher thread {Central}] |common.TezUtilsInternal|: Redirecting log file based on addend: dag_1611750110647_64528_1_post End of LogType:syslog_dag_1611750110647_64528_1 ************************************************************************************************* Container: container_e703_1611750110647_64528_01_000001 on wx12-dsj-node045_45454_1611898349701 LogAggregationType: AGGREGATED =============================================================================================== LogType:syslog_dag_1611750110647_64528_1_post LogLastModifiedTime:Fri Jan 29 13:32:30 +0800 2021 LogLength:7096 LogContents: 2021-01-29 13:32:20,579 [INFO] [HistoryEventHandlingThread] |compress.CodecPool|: Got brand-new compressor [.deflate] 2021-01-29 13:32:20,873 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Central Dispatcher queue size after DAG completion, before cleanup: 0 2021-01-29 13:32:20,712 [INFO] [IPC Server handler 1 on 17961] |client.DAGClientHandler|: Received message to shutdown AM from data_quality (auth:SIMPLE) at 172.22.1.72 2021-01-29 13:32:20,873 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Waiting for next DAG to be submitted. 2021-01-29 13:32:20,875 [INFO] [IPC Server handler 1 on 17961] |rm.TaskSchedulerManager|: TaskScheduler notified that it should unregister from RM 2021-01-29 13:32:20,875 [INFO] [IPC Server handler 1 on 17961] |app.DAGAppMaster|: No current running DAG, shutting down the AM 2021-01-29 13:32:20,875 [INFO] [IPC Server handler 1 on 17961] |app.DAGAppMaster|: DAGAppMasterShutdownHandler invoked 2021-01-29 13:32:20,875 [INFO] [IPC Server handler 1 on 17961] |app.DAGAppMaster|: Handling DAGAppMaster shutdown 2021-01-29 13:32:20,876 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Cleaning up DAG: name=select count(1) as counts, 0 a...'11' (Stage-1), with id=dag_1611750110647_64528_1 2021-01-29 13:32:20,876 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Sleeping for 5000 ms before shutting down 2021-01-29 13:32:20,877 [INFO] [Dispatcher thread {Central}] |container.AMContainerMap|: Cleaned up completed containers on dagComplete. Removed=0, Remaining=1 2021-01-29 13:32:20,877 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Completed cleanup for DAG: name=select count(1) as counts, 0 a...'11' (Stage-1), with id=dag_1611750110647_64528_1 2021-01-29 13:32:23,142 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: Free: pendingRequests: 0 delayedContainers: 1 heartbeats: 51 lastPreemptionHeartbeat: 50 2021-01-29 13:32:25,877 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Calling stop for all the services 2021-01-29 13:32:25,878 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Initiating stop of YarnTaskScheduler 2021-01-29 13:32:25,878 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Releasing held containers 2021-01-29 13:32:25,883 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Removing all pending taskRequests 2021-01-29 13:32:25,886 [INFO] [AMShutdownThread] |history.HistoryEventHandler|: Stopping HistoryEventHandler 2021-01-29 13:32:25,887 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Stopping RecoveryService 2021-01-29 13:32:25,888 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Handle the remaining events in queue, queue size=0 2021-01-29 13:32:25,888 [INFO] [RecoveryEventHandlingThread] |recovery.RecoveryService|: EventQueue take interrupted. Returning 2021-01-29 13:32:25,889 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Closing Summary Stream 2021-01-29 13:32:25,907 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1611750110647_64528_1][Event:CONTAINER_STOPPED]: containerId=container_e703_1611750110647_64528_01_000002, stoppedTime=1611898345905, exitStatus=0 2021-01-29 13:32:25,924 [INFO] [ContainerLauncher #1] |launcher.TezContainerLauncherImpl|: Stopping container_e703_1611750110647_64528_01_000002 2021-01-29 13:32:25,994 [INFO] [IPC Server handler 13 on 27287] |app.TezTaskCommunicatorImpl|: Container with id: container_e703_1611750110647_64528_01_000002 is valid, but no longer registered, and will be killed 2021-01-29 13:32:26,010 [WARN] [AMShutdownThread] |recovery.RecoveryService|: Ignoring error while closing summary stream. The recovery directory at hdfs://ztcluster/tmp/hive/data_quality/_tez_session_dir/74300ba4-ffcf-43e7-87c4-a2ea8a280d1c/.tez/application_1611750110647_64528/recovery/1 has already been deleted externally 2021-01-29 13:32:26,029 [INFO] [AMShutdownThread] |proto.ProtoHistoryLoggingService|: Stopping ProtoHistoryLoggingService, eventQueueBacklog=0 2021-01-29 13:32:26,047 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: Skipping delayed container as container is no longer running, containerId=container_e703_1611750110647_64528_01_000002 2021-01-29 13:32:26,151 [INFO] [AMShutdownThread] |proto.ProtoHistoryLoggingService|: Stopped ProtoHistoryLoggingService 2021-01-29 13:32:26,152 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: AllocatedContainerManager Thread interrupted 2021-01-29 13:32:26,154 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Unregistering application from RM, exitStatus=UNDEFINED, exitMessage=Session stats:submittedDAGs=1, successfulDAGs=1, failedDAGs=0, killedDAGs=0 , trackingURL=http://hdp-hive-server.s.2345inc.com:9999/tez-ui/?viewPath=%2F%23%2Ftez-app%2Fapplication_1611750110647_64528 2021-01-29 13:32:26,158 [INFO] [AMShutdownThread] |impl.AMRMClientImpl|: Waiting for application to be successfully unregistered. 2021-01-29 13:32:26,261 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Successfully unregistered application from RM 2021-01-29 13:32:26,262 [INFO] [AMShutdownThread] |rm.TaskSchedulerManager|: Shutting down AppCallbackExecutor 2021-01-29 13:32:26,269 [INFO] [AMShutdownThread] |handler.ContextHandler|: Stopped o.e.j.w.WebAppContext@614cc5f0{/,null,UNAVAILABLE}{/} 2021-01-29 13:32:26,274 [INFO] [AMShutdownThread] |server.AbstractConnector|: Stopped ServerConnector@463f922f{HTTP/1.1,[http/1.1]}{0.0.0.0:0} 2021-01-29 13:32:26,275 [INFO] [AMShutdownThread] |handler.ContextHandler|: Stopped o.e.j.s.ServletContextHandler@1b6984ef{/static,jar:file:/data01/hdp/yarn/local/filecache/2160/tez.tar.gz/lib/hadoop-yarn-common-3.1.1.3.1.5.0-152.jar!/webapps/static,UNAVAILABLE} 2021-01-29 13:32:26,276 [INFO] [AMShutdownThread] |ipc.Server|: Stopping server on 27287 2021-01-29 13:32:26,277 [INFO] [IPC Server listener on 27287] |ipc.Server|: Stopping IPC Server listener on 27287 2021-01-29 13:32:26,277 [INFO] [IPC Server Responder] |ipc.Server|: Stopping IPC Server Responder 2021-01-29 13:32:26,277 [INFO] [AMShutdownThread] |ipc.Server|: Stopping server on 17961 2021-01-29 13:32:26,278 [INFO] [IPC Server listener on 17961] |ipc.Server|: Stopping IPC Server listener on 17961 2021-01-29 13:32:26,278 [INFO] [IPC Server Responder] |ipc.Server|: Stopping IPC Server Responder 2021-01-29 13:32:26,282 [INFO] [shutdown-hook-0] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked 2021-01-29 13:32:26,282 [INFO] [shutdown-hook-0] |app.DAGAppMaster|: The shutdown handler is still running, waiting for it to complete 2021-01-29 13:32:26,286 [WARN] [AMShutdownThread] |app.DAGAppMaster|: Failed to delete tez scratch data dir, path=hdfs://ztcluster/tmp/hive/data_quality/_tez_session_dir/74300ba4-ffcf-43e7-87c4-a2ea8a280d1c/.tez/application_1611750110647_64528 2021-01-29 13:32:26,286 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Exiting DAGAppMaster..GoodBye! 2021-01-29 13:32:26,286 [INFO] [shutdown-hook-0] |app.DAGAppMaster|: The shutdown handler has completed End of LogType:syslog_dag_1611750110647_64528_1_post ******************************************************************************************************