================================================================= >>> Invoking Main class now >>> Fetching child yarn jobs tag id : oozie-b520d7b146cddb5242b38e05c43b8b68 Child yarn jobs are found - 2018-04-17 17:16:19,269 [main] WARN org.apache.hadoop.hdfs.ClientContext - Existing client context 'default' does not match requested configuration. Existing: shortCircuitStreamsCacheSize = 256, shortCircuitStreamsCacheExpiryMs = 300000, shortCircuitMmapCacheSize = 256, shortCircuitMmapCacheExpiryMs = 3600000, shortCircuitMmapCacheRetryTimeout = 300000, shortCircuitCacheStaleThresholdMs = 1800000, socketCacheCapacity = 16, socketCacheExpiry = 3000, shortCircuitLocalReads = false, useLegacyBlockReaderLocal = false, domainSocketDataTraffic = false, shortCircuitSharedMemoryWatcherInterruptCheckMs = 60000, keyProviderCacheExpiryMs = 864000000, Requested: shortCircuitStreamsCacheSize = 1024, shortCircuitStreamsCacheExpiryMs = 300000, shortCircuitMmapCacheSize = 256, shortCircuitMmapCacheExpiryMs = 3600000, shortCircuitMmapCacheRetryTimeout = 300000, shortCircuitCacheStaleThresholdMs = 1800000, socketCacheCapacity = 16, socketCacheExpiry = 3000, shortCircuitLocalReads = true, useLegacyBlockReaderLocal = false, domainSocketDataTraffic = false, shortCircuitSharedMemoryWatcherInterruptCheckMs = 60000, keyProviderCacheExpiryMs = 864000000 Spark Action Main class : org.apache.spark.deploy.SparkSubmit Oozie Spark action configuration ================================================================= --master yarn-cluster --name com.jd.aof.rigel --class com.jd.aof.rigel.Rigel --executor-memory 6g --num-executors 800 --driver-memory 20g --executor-cores 2 --files hdfs://ns2:8020/user/testoozie/rigel-pipe/28/oozie/jobs/engine/rigel/configs/20180327.xml --queue default --conf spark.yarn.historyServer.address=http://172.16.172.95:18080 --conf spark.yarn.archive=hdfs://ns2/user/spark/lib/spark-2.1.0-hadoop-2.7.1.zip --conf spark.eventLog.dir=hdfs://ns2/user/spark/log --conf spark.eventLog.enabled=true --conf spark.executor.extraClassPath=$PWD/* --conf spark.driver.extraClassPath=$PWD/* --conf spark.yarn.tags=oozie-b520d7b146cddb5242b38e05c43b8b68 --conf spark.yarn.security.tokens.hive.enabled=false --conf spark.yarn.security.tokens.hbase.enabled=false --conf spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties --conf spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties --conf spark.yarn.jar=null --verbose hdfs://ns2:8020/user/testoozie/rigel-pipe/28/oozie/jobs/engine/rigel/rigel.jar --input hdfs://ns2:8020/user/mart_risk_aof/projects/aof/feeds//aof_features_decorator/daily/20180415 --output hdfs://ns2:8020/user/testresults//rigel/json/daily/20180415 --output-format gzip --config-path 20180327.xml ================================================================= >>> Invoking Spark class now >>> 2018-04-17 17:16:20,163 [main] WARN org.apache.spark.SparkConf - The configuration key 'spark.yarn.jar' has been deprecated as of Spark 2.0 and may be removed in the future. Please use the new key 'spark.yarn.jars' instead. 2018-04-17 17:16:20,454 [main] WARN org.apache.spark.deploy.yarn.security.ConfigurableCredentialManager - spark.yarn.security.tokens.hbase.enabled is deprecated, using spark.yarn.security.credentials.hbase.enabled instead 2018-04-17 17:16:20,456 [main] WARN org.apache.spark.deploy.yarn.security.ConfigurableCredentialManager - spark.yarn.security.tokens.hive.enabled is deprecated, using spark.yarn.security.credentials.hive.enabled instead 2018-04-17 17:16:20,518 [main] INFO org.apache.spark.deploy.yarn.Client - Requesting a new application from cluster with 4671 NodeManagers 2018-04-17 17:16:20,534 [main] INFO org.apache.spark.deploy.yarn.Client - Verifying our application has not requested more than the maximum memory capability of the cluster (28672 MB per container) 2018-04-17 17:16:20,534 [main] INFO org.apache.spark.deploy.yarn.Client - Will allocate AM container, with 22528 MB memory including 2048 MB overhead 2018-04-17 17:16:20,535 [main] INFO org.apache.spark.deploy.yarn.Client - Setting up container launch context for our AM 2018-04-17 17:16:20,536 [main] INFO org.apache.spark.deploy.yarn.Client - Setting up the launch environment for our AM container 2018-04-17 17:16:20,571 [main] INFO org.apache.spark.deploy.yarn.Client - Preparing resources for our AM container 2018-04-17 17:16:20,643 [main] INFO org.apache.spark.deploy.yarn.Client - Uploading resource hdfs://ns2/user/spark/lib/spark-2.1.0-hadoop-2.7.1.zip -> hdfs://ns10/user/test/.sparkStaging/application_1523860188086_164713/spark-2.1.0-hadoop-2.7.1.zip 2018-04-17 17:16:25,355 [main] INFO org.apache.spark.deploy.yarn.Client - Uploading resource hdfs://ns2:8020/user/testoozie/rigel-pipe/28/oozie/jobs/engine/rigel/rigel.jar -> hdfs://ns10/user/test/.sparkStaging/application_1523860188086_164713/rigel.jar 2018-04-17 17:16:25,807 [main] INFO org.apache.spark.deploy.yarn.Client - Uploading resource hdfs://ns2:8020/user/testoozie/rigel-pipe/28/oozie/jobs/engine/rigel/configs/20180327.xml -> hdfs://ns10/user/test/.sparkStaging/application_1523860188086_164713/20180327.xml 2018-04-17 17:16:26,431 [main] INFO org.apache.spark.deploy.yarn.Client - Uploading resource file:/data0/yarn1/local/usercache/test/appcache/application_1523860188086_164609/spark-7f861f4e-e9dc-4def-afe3-8d338fdcc166/__spark_conf__3846017792796170805.zip -> hdfs://ns10/user/test/.sparkStaging/application_1523860188086_164713/__spark_conf__.zip 2018-04-17 17:16:26,740 [main] INFO org.apache.spark.SecurityManager - Changing view acls to: yarn,mart_risk_aof 2018-04-17 17:16:26,741 [main] INFO org.apache.spark.SecurityManager - Changing modify acls to: yarn,mart_risk_aof 2018-04-17 17:16:26,741 [main] INFO org.apache.spark.SecurityManager - Changing view acls groups to: 2018-04-17 17:16:26,742 [main] INFO org.apache.spark.SecurityManager - Changing modify acls groups to: 2018-04-17 17:16:26,748 [main] INFO org.apache.spark.SecurityManager - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, mart_risk_aof); groups with view permissions: Set(); users with modify permissions: Set(yarn, mart_risk_aof); groups with modify permissions: Set() 2018-04-17 17:16:26,769 [main] INFO org.apache.spark.deploy.yarn.Client - Submitting application application_1523860188086_164713 to ResourceManager 2018-04-17 17:16:28,634 [main] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Application submission is not finished, submitted application application_1523860188086_164713 is still in SUBMITTED 2018-04-17 17:16:30,648 [main] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Application submission is not finished, submitted application application_1523860188086_164713 is still in SUBMITTED 2018-04-17 17:16:32,663 [main] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Application submission is not finished, submitted application application_1523860188086_164713 is still in SUBMITTED 2018-04-17 17:16:34,673 [main] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Application submission is not finished, submitted application application_1523860188086_164713 is still in SUBMITTED 2018-04-17 17:16:35,880 [main] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1523860188086_164713 2018-04-17 17:16:36,885 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:36,892 [main] INFO org.apache.spark.deploy.yarn.Client - client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.default start time: 1523956586843 final status: UNDEFINED tracking URL: http://resourcemanager:8088/proxy/application_1523860188086_164713/ user: mart_risk_aof 2018-04-17 17:16:37,895 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:38,896 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:39,898 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:40,900 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:41,901 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:42,903 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:43,904 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:44,906 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:45,907 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:46,909 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:47,912 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) Heart beat 2018-04-17 17:16:48,914 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:49,915 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:50,916 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:51,918 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:52,921 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:53,922 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:54,924 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:55,927 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:56,928 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:57,930 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:58,931 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:16:59,932 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:00,934 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:01,935 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:02,937 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:03,938 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:04,940 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:05,941 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:06,943 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:07,944 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:08,946 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:09,949 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:10,951 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:11,954 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:12,955 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:13,956 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:14,958 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:15,960 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:16,961 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED) 2018-04-17 17:17:17,963 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1523860188086_164713 (state: ACCEPTED)