Container: container_e02_1502070770869_0012_01_000003 on host4_25454_1502075200681 LogAggregationType: AGGREGATED ============================================================================================================================ LogType:directory.info Log Upload Time:Mon Aug 07 03:06:40 +0000 2017 LogLength:12207 Log Contents: ls -l: total 12 -rw------- 1 hrt_qa hadoop 633 Aug 7 03:06 container_tokens -rwx------ 1 hrt_qa hadoop 7666 Aug 7 03:06 launch_container.sh lrwxrwxrwx 1 hrt_qa hadoop 60 Aug 7 03:06 ml.R -> /grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/15/ml.R lrwxrwxrwx 1 hrt_qa hadoop 93 Aug 7 03:06 __spark_conf__ -> /grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/16/__spark_conf__5875502619868816278.zip lrwxrwxrwx 1 hrt_qa hadoop 61 Aug 7 03:06 __spark__.jar -> /grid/0/hadoop/yarn/local/filecache/12/spark-hdp-assembly.jar lrwxrwxrwx 1 hrt_qa hadoop 66 Aug 7 03:06 sparkr -> /grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/14/sparkr.zip drwxr-s--- 2 hrt_qa hadoop 6 Aug 7 03:06 tmp find -L . -maxdepth 5 -ls: 540117076 0 drwxr-s--- 3 hrt_qa hadoop 130 Aug 7 03:06 . 676281527 0 drwxr-s--- 2 hrt_qa hadoop 6 Aug 7 03:06 ./tmp 540117077 8 -rwx------ 1 hrt_qa hadoop 7666 Aug 7 03:06 ./launch_container.sh 540117078 4 -rw------- 1 hrt_qa hadoop 633 Aug 7 03:06 ./container_tokens 1010802710 0 drwx------ 3 hrt_qa hrt_qa 19 Aug 7 03:06 ./sparkr 1747995 0 drwx------ 10 hrt_qa hrt_qa 139 Aug 7 03:06 ./sparkr/SparkR 1747996 8 -r-x------ 1 hrt_qa hrt_qa 7018 Aug 7 03:06 ./sparkr/SparkR/NAMESPACE 70865960 0 drwx------ 2 hrt_qa hrt_qa 88 Aug 7 03:06 ./sparkr/SparkR/Meta 70865961 4 -r-x------ 1 hrt_qa hrt_qa 164 Aug 7 03:06 ./sparkr/SparkR/Meta/Rd.rds 70865962 4 -r-x------ 1 hrt_qa hrt_qa 188 Aug 7 03:06 ./sparkr/SparkR/Meta/hsearch.rds 70865963 4 -r-x------ 1 hrt_qa hrt_qa 1717 Aug 7 03:06 ./sparkr/SparkR/Meta/nsInfo.rds 70865964 4 -r-x------ 1 hrt_qa hrt_qa 746 Aug 7 03:06 ./sparkr/SparkR/Meta/package.rds 70865965 4 -r-x------ 1 hrt_qa hrt_qa 38 Aug 7 03:06 ./sparkr/SparkR/Meta/links.rds 144779287 0 drwx------ 2 hrt_qa hrt_qa 38 Aug 7 03:06 ./sparkr/SparkR/test_support 144779288 4 -r-x------ 1 hrt_qa hrt_qa 2886 Aug 7 03:06 ./sparkr/SparkR/test_support/sparktestjar_2.10-1.0.jar 205317228 0 drwx------ 2 hrt_qa hrt_qa 88 Aug 7 03:06 ./sparkr/SparkR/help 205317229 4 -r-x------ 1 hrt_qa hrt_qa 64 Aug 7 03:06 ./sparkr/SparkR/help/paths.rds 205317230 0 -r-x------ 1 hrt_qa hrt_qa 0 Aug 7 03:06 ./sparkr/SparkR/help/SparkR.rdb 205317231 4 -r-x------ 1 hrt_qa hrt_qa 60 Aug 7 03:06 ./sparkr/SparkR/help/aliases.rds 205317232 0 -r-x------ 1 hrt_qa hrt_qa 0 Aug 7 03:06 ./sparkr/SparkR/help/AnIndex 205317233 4 -r-x------ 1 hrt_qa hrt_qa 113 Aug 7 03:06 ./sparkr/SparkR/help/SparkR.rdx 271809573 0 drwx------ 3 hrt_qa hrt_qa 21 Aug 7 03:06 ./sparkr/SparkR/tests 340968465 4 drwx------ 2 hrt_qa hrt_qa 4096 Aug 7 03:06 ./sparkr/SparkR/tests/testthat 340968505 28 -r-x------ 1 hrt_qa hrt_qa 27490 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_rdd.R 340968506 4 -r-x------ 1 hrt_qa hrt_qa 1957 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_client.R 340968507 4 -r-x------ 1 hrt_qa hrt_qa 3981 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_context.R 340968508 4 -r-x------ 1 hrt_qa hrt_qa 3691 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_binary_function.R 340968509 4 -r-x------ 1 hrt_qa hrt_qa 3171 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_take.R 340968510 8 -r-x------ 1 hrt_qa hrt_qa 5137 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_textFile.R 340968511 8 -r-x------ 1 hrt_qa hrt_qa 4380 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_parallelize_collect.R 340967452 8 -r-x------ 1 hrt_qa hrt_qa 4859 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_mllib.R 340912155 4 -r-x------ 1 hrt_qa hrt_qa 1241 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/jarTest.R 340912161 4 -r-x------ 1 hrt_qa hrt_qa 1542 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_includeJAR.R 340912162 4 -r-x------ 1 hrt_qa hrt_qa 3064 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_binaryFile.R 340430848 4 -r-x------ 1 hrt_qa hrt_qa 1611 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_broadcast.R 340430851 4 -r-x------ 1 hrt_qa hrt_qa 1810 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_includePackage.R 340432896 4 -r-x------ 1 hrt_qa hrt_qa 2327 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_Serde.R 340432897 4 -r-x------ 1 hrt_qa hrt_qa 996 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/packageInAJarTest.R 340432898 8 -r-x------ 1 hrt_qa hrt_qa 5201 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_utils.R 340432899 8 -r-x------ 1 hrt_qa hrt_qa 7922 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_shuffle.R 340432900 72 -r-x------ 1 hrt_qa hrt_qa 70877 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_sparkSQL.R 1747997 4 -r-x------ 1 hrt_qa hrt_qa 678 Aug 7 03:06 ./sparkr/SparkR/DESCRIPTION 403415064 0 drwx------ 2 hrt_qa hrt_qa 37 Aug 7 03:06 ./sparkr/SparkR/html 403415065 4 -r-x------ 1 hrt_qa hrt_qa 923 Aug 7 03:06 ./sparkr/SparkR/html/00Index.html 403415066 4 -r-x------ 1 hrt_qa hrt_qa 1319 Aug 7 03:06 ./sparkr/SparkR/html/R.css 472623140 0 drwx------ 2 hrt_qa hrt_qa 53 Aug 7 03:06 ./sparkr/SparkR/R 472623141 4 -r-x------ 1 hrt_qa hrt_qa 1056 Aug 7 03:06 ./sparkr/SparkR/R/SparkR 472623142 804 -r-x------ 1 hrt_qa hrt_qa 821006 Aug 7 03:06 ./sparkr/SparkR/R/SparkR.rdb 472653849 20 -r-x------ 1 hrt_qa hrt_qa 19182 Aug 7 03:06 ./sparkr/SparkR/R/SparkR.rdx 540117073 0 drwx------ 2 hrt_qa hrt_qa 36 Aug 7 03:06 ./sparkr/SparkR/profile 540117074 4 -r-x------ 1 hrt_qa hrt_qa 1696 Aug 7 03:06 ./sparkr/SparkR/profile/shell.R 540117075 4 -r-x------ 1 hrt_qa hrt_qa 958 Aug 7 03:06 ./sparkr/SparkR/profile/general.R 608539664 0 drwx------ 2 hrt_qa hrt_qa 36 Aug 7 03:06 ./sparkr/SparkR/worker 608539665 8 -r-x------ 1 hrt_qa hrt_qa 5884 Aug 7 03:06 ./sparkr/SparkR/worker/worker.R 608539666 4 -r-x------ 1 hrt_qa hrt_qa 1863 Aug 7 03:06 ./sparkr/SparkR/worker/daemon.R 875621379 186408 -r-xr-xr-x 1 yarn hadoop 190878097 Aug 7 02:35 ./__spark__.jar 876876813 4 drwx------ 2 hrt_qa hrt_qa 4096 Aug 7 03:06 ./__spark_conf__ 876876814 4 -r-x------ 1 hrt_qa hrt_qa 620 Aug 7 03:06 ./__spark_conf__/log4j.properties 876876815 8 -r-x------ 1 hrt_qa hrt_qa 6606 Aug 7 03:06 ./__spark_conf__/hadoop-env.sh 876876816 8 -r-x------ 1 hrt_qa hrt_qa 7831 Aug 7 03:06 ./__spark_conf__/mapred-site.xml 876876817 4 -r-x------ 1 hrt_qa hrt_qa 3085 Aug 7 03:06 ./__spark_conf__/hadoop-metrics2.properties 876876818 28 -r-x------ 1 hrt_qa hrt_qa 24749 Aug 7 03:06 ./__spark_conf__/yarn-site.xml 876876819 4 -r-x------ 1 hrt_qa hrt_qa 3979 Aug 7 03:06 ./__spark_conf__/hadoop-env.cmd 876876820 8 -r-x------ 1 hrt_qa hrt_qa 7596 Aug 7 03:06 ./__spark_conf__/core-site.xml 876876821 0 -r-x------ 1 hrt_qa hrt_qa 0 Aug 7 03:06 ./__spark_conf__/yarn.exclude 876876822 4 -r-x------ 1 hrt_qa hrt_qa 1 Aug 7 03:06 ./__spark_conf__/dfs.exclude 876876823 4 -r-x------ 1 hrt_qa hrt_qa 1631 Aug 7 03:06 ./__spark_conf__/kms-log4j.properties 876876824 4 -r-x------ 1 hrt_qa hrt_qa 2250 Aug 7 03:06 ./__spark_conf__/yarn-env.cmd 876876825 4 -r-x------ 1 hrt_qa hrt_qa 917 Aug 7 03:06 ./__spark_conf__/ssl-client.xml 876876826 4 -r-x------ 1 hrt_qa hrt_qa 2313 Aug 7 03:06 ./__spark_conf__/capacity-scheduler.xml 876876827 4 -r-x------ 1 hrt_qa hrt_qa 2610 Aug 7 03:06 ./__spark_conf__/ranger-hdfs-audit.xml 876876828 4 -r-x------ 1 hrt_qa hrt_qa 3518 Aug 7 03:06 ./__spark_conf__/kms-acls.xml 876876829 4 -r-x------ 1 hrt_qa hrt_qa 758 Aug 7 03:06 ./__spark_conf__/mapred-site.xml.template 876876830 4 -r-x------ 1 hrt_qa hrt_qa 2358 Aug 7 03:06 ./__spark_conf__/topology_script.py 876876831 4 -r-x------ 1 hrt_qa hrt_qa 333 Aug 7 03:06 ./__spark_conf__/yarn_nm_jaas.conf 876876832 4 -r-x------ 1 hrt_qa hrt_qa 1335 Aug 7 03:06 ./__spark_conf__/configuration.xsl 876876833 4 -r-x------ 1 hrt_qa hrt_qa 64 Aug 7 03:06 ./__spark_conf__/ranger-security.xml 876876834 8 -r-x------ 1 hrt_qa hrt_qa 5532 Aug 7 03:06 ./__spark_conf__/yarn-env.sh 876876835 12 -r-x------ 1 hrt_qa hrt_qa 11405 Aug 7 03:06 ./__spark_conf__/hdfs-site.xml 876876836 4 -r-x------ 1 hrt_qa hrt_qa 337 Aug 7 03:06 ./__spark_conf__/yarn_ats_jaas.conf 876876837 4 -r-x------ 1 hrt_qa hrt_qa 1020 Aug 7 03:06 ./__spark_conf__/commons-logging.properties 876876838 4 -r-x------ 1 hrt_qa hrt_qa 333 Aug 7 03:06 ./__spark_conf__/hdfs_nn_jaas.conf 876876839 4 -r-x------ 1 hrt_qa hrt_qa 333 Aug 7 03:06 ./__spark_conf__/hdfs_dn_jaas.conf 876876840 4 -r-x------ 1 hrt_qa hrt_qa 1033 Aug 7 03:06 ./__spark_conf__/container-executor.cfg 876876841 4 -r-x------ 1 hrt_qa hrt_qa 571 Aug 7 03:06 ./__spark_conf__/yarn_jaas.conf 876876842 8 -r-x------ 1 hrt_qa hrt_qa 4221 Aug 7 03:06 ./__spark_conf__/task-log4j.properties 876876843 4 -r-x------ 1 hrt_qa hrt_qa 2490 Aug 7 03:06 ./__spark_conf__/hadoop-metrics.properties 876876844 4 -r-x------ 1 hrt_qa hrt_qa 984 Aug 7 03:06 ./__spark_conf__/mapred-env.sh 876876845 4 -r-x------ 1 hrt_qa hrt_qa 1602 Aug 7 03:06 ./__spark_conf__/health_check 876876846 4 -r-x------ 1 hrt_qa hrt_qa 1001 Aug 7 03:06 ./__spark_conf__/ranger-policymgr-ssl-yarn.xml 876876847 4 -r-x------ 1 hrt_qa hrt_qa 320 Aug 7 03:06 ./__spark_conf__/mapred_jaas.conf 876876848 4 -r-x------ 1 hrt_qa hrt_qa 2316 Aug 7 03:06 ./__spark_conf__/ssl-client.xml.example 876876849 4 -r-x------ 1 hrt_qa hrt_qa 1527 Aug 7 03:06 ./__spark_conf__/kms-env.sh 876876850 4 -r-x------ 1 hrt_qa hrt_qa 1308 Aug 7 03:06 ./__spark_conf__/hadoop-policy.xml 876876851 4 -r-x------ 1 hrt_qa hrt_qa 2610 Aug 7 03:06 ./__spark_conf__/ranger-yarn-audit.xml 876876852 4 -r-x------ 1 hrt_qa hrt_qa 145 Aug 7 03:06 ./__spark_conf__/slaves 876876853 4 -r-x------ 1 hrt_qa hrt_qa 287 Aug 7 03:06 ./__spark_conf__/topology_mappings.data 876876854 4 -r-x------ 1 hrt_qa hrt_qa 1049 Aug 7 03:06 ./__spark_conf__/ranger-hdfs-security.xml 876876855 8 -r-x------ 1 hrt_qa hrt_qa 4956 Aug 7 03:06 ./__spark_conf__/metrics.properties 876876856 4 -r-x------ 1 hrt_qa hrt_qa 951 Aug 7 03:06 ./__spark_conf__/mapred-env.cmd 876876857 4 -r-x------ 1 hrt_qa hrt_qa 1054 Aug 7 03:06 ./__spark_conf__/ssl-server.xml 876876858 4 -r-x------ 1 hrt_qa hrt_qa 333 Aug 7 03:06 ./__spark_conf__/hdfs_jn_jaas.conf 876876859 4 -r-x------ 1 hrt_qa hrt_qa 2697 Aug 7 03:06 ./__spark_conf__/ssl-server.xml.example 876876860 4 -r-x------ 1 hrt_qa hrt_qa 995 Aug 7 03:06 ./__spark_conf__/ranger-policymgr-ssl.xml 876876861 4 -r-x------ 1 hrt_qa hrt_qa 935 Aug 7 03:06 ./__spark_conf__/ranger-yarn-security.xml 876876862 4 -r-x------ 1 hrt_qa hrt_qa 946 Aug 7 03:06 ./__spark_conf__/taskcontroller.cfg 876876863 8 -r-x------ 1 hrt_qa hrt_qa 4113 Aug 7 03:06 ./__spark_conf__/mapred-queues.xml.template 875621406 8 -r-x------ 1 hrt_qa hrt_qa 5511 Aug 7 03:06 ./__spark_conf__/kms-site.xml 875621407 4 -r-x------ 1 hrt_qa hrt_qa 1661 Aug 7 03:06 ./__spark_conf__/__spark_conf__.properties 676291710 4 -r-x------ 1 hrt_qa hrt_qa 1903 Aug 7 03:06 ./ml.R broken symlinks(find -L . -maxdepth 5 -type l -ls): End of LogType:directory.info LogType:launch_container.sh Log Upload Time:Mon Aug 07 03:06:40 +0000 2017 LogLength:7666 Log Contents: #!/bin/bash export SPARK_YARN_STAGING_DIR=".sparkStaging/application_1502070770869_0012" export PATH="/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/var/lib/ambari-agent" export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/hdp/current/hadoop-client/conf"} export JAVA_HOME=${JAVA_HOME:-"/usr/lib/jvm/java-openjdk"} export SPARK_YARN_CACHE_FILES="hdfs://mycluster/hdp/apps/2.6.2.0-154/spark/spark-hdp-assembly.jar#__spark__.jar,hdfs://mycluster/user/hrt_qa/.sparkStaging/application_1502070770869_0012/ml.R#ml.R" export LANG="en_US.UTF-8" export SPARK_LOG_URL_STDOUT="https://host4:54321/node/containerlogs/container_e02_1502070770869_0012_01_000003/hrt_qa/stdout?start=-4096" export NM_HOST="host4" export SPARK_YARN_CACHE_FILES_FILE_SIZES="190878097,1903" export SPARK_YARN_CACHE_ARCHIVES_TIME_STAMPS="1502075165966,1502075166070" export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/lib64/R/lib:/usr/lib/jvm/jre/lib/amd64/server:/usr/lib/jvm/jre/lib/amd64:/usr/lib/jvm/java/lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib:/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:" export LOGNAME="hrt_qa" export JVM_PID="$$" export PWD="/grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1502070770869_0012/container_e02_1502070770869_0012_01_000003" export LOCAL_DIRS="/grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1502070770869_0012" export NM_HTTP_PORT="54321" export LOG_DIRS="/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000003" export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= " export NM_PORT="25454" export SPARK_YARN_CACHE_FILES_TIME_STAMPS="1502064724925,1502075165821" export USER="hrt_qa" export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/hdp/current/hadoop-yarn-nodemanager"} export CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark__.jar:/etc/hadoop/conf:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:/usr/hdp/current/ext/hadoop/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/2.6.2.0-154/hadoop/lib/hadoop-lzo-0.6.0.2.6.2.0-154.jar:/etc/hadoop/conf/secure:/usr/hdp/current/ext/hadoop/*" export SPARK_YARN_CACHE_ARCHIVES="hdfs://mycluster/user/hrt_qa/.sparkStaging/application_1502070770869_0012/sparkr.zip#sparkr,hdfs://mycluster/user/hrt_qa/.sparkStaging/application_1502070770869_0012/__spark_conf__5875502619868816278.zip#__spark_conf__" export SPARK_YARN_CACHE_ARCHIVES_FILE_SIZES="555499,144034" export SPARK_YARN_MODE="true" export SPARK_YARN_CACHE_FILES_VISIBILITIES="PUBLIC,PRIVATE" export HADOOP_TOKEN_FILE_LOCATION="/grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1502070770869_0012/container_e02_1502070770869_0012_01_000003/container_tokens" export NM_AUX_SERVICE_spark_shuffle="" export SPARK_USER="hrt_qa" export LOCAL_USER_DIRS="/grid/0/hadoop/yarn/local/usercache/hrt_qa/" export HADOOP_HOME="/usr/hdp/2.6.2.0-154/hadoop" export SPARK_LOG_URL_STDERR="https://host4:54321/node/containerlogs/container_e02_1502070770869_0012_01_000003/hrt_qa/stderr?start=-4096" export SPARK_YARN_CACHE_ARCHIVES_VISIBILITIES="PRIVATE,PRIVATE" export HOME="/home/" export NM_AUX_SERVICE_spark2_shuffle="" export CONTAINER_ID="container_e02_1502070770869_0012_01_000003" export MALLOC_ARENA_MAX="4" ln -sf "/grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/14/sparkr.zip" "sparkr" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/grid/0/hadoop/yarn/local/filecache/12/spark-hdp-assembly.jar" "__spark__.jar" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/16/__spark_conf__5875502619868816278.zip" "__spark_conf__" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/15/ml.R" "ml.R" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi # Creating copy of launch script cp "launch_container.sh" "/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000003/launch_container.sh" chmod 640 "/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000003/launch_container.sh" # Determining directory contents echo "ls -l:" 1>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000003/directory.info" ls -l 1>>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000003/directory.info" echo "find -L . -maxdepth 5 -ls:" 1>>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000003/directory.info" find -L . -maxdepth 5 -ls 1>>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000003/directory.info" echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000003/directory.info" find -L . -maxdepth 5 -type l -ls 1>>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000003/directory.info" exec /bin/bash -c "LD_LIBRARY_PATH="/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms1024m -Xmx1024m -Djava.io.tmpdir=$PWD/tmp '-Dspark.authenticate=true' '-Dspark.ssl.trustStore=/etc/security/clientKeys/spark_all.jks' '-Dspark.ssl.keyStore=/etc/security/serverKeys/spark_keystore.jks' '-Dspark.authenticate.enableSaslEncryption=true' '-Dspark.driver.port=53417' '-Dspark.ssl.enabledAlgorithms=TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA' '-Dspark.ssl.enabled=true' '-Dspark.history.ui.port=18080' '-Dspark.ssl.trustStorePassword=clientTrustStorePassword' '-Dspark.ui.port=0' '-Dspark.ssl.protocol=TLS' '-Dspark.ssl.keyPassword=serverKeyPassPassword' '-Dspark.ssl.keyStorePassword=serverStorePassPassword' -Dspark.yarn.app.container.log.dir=/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000003 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@xxx.xx.xx.xxx:53417 --executor-id 2 --hostname host4 --cores 1 --app-id application_1502070770869_0012 --user-class-path file:$PWD/__app__.jar 1> /grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000003/stdout 2> /grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000003/stderr" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi End of LogType:launch_container.sh LogType:stderr Log Upload Time:Mon Aug 07 03:06:40 +0000 2017 LogLength:39256 Log Contents: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/grid/0/hadoop/yarn/local/filecache/12/spark-hdp-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.2.0-154/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 17/08/07 03:06:18 INFO CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT] 17/08/07 03:06:18 INFO SecurityManager: Changing view acls to: hrt_qa 17/08/07 03:06:18 INFO SecurityManager: Changing modify acls to: hrt_qa 17/08/07 03:06:18 INFO SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: Set(hrt_qa); users with modify permissions: Set(hrt_qa) 17/08/07 03:06:19 INFO SecurityManager: Changing view acls to: hrt_qa 17/08/07 03:06:19 INFO SecurityManager: Changing modify acls to: hrt_qa 17/08/07 03:06:19 INFO SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: Set(hrt_qa); users with modify permissions: Set(hrt_qa) 17/08/07 03:06:19 INFO Slf4jLogger: Slf4jLogger started 17/08/07 03:06:19 INFO Remoting: Starting remoting 17/08/07 03:06:19 INFO Remoting: Remoting started; listening on addresses :[akka.ssl.tcp://sparkExecutorActorSystem@host4:39229] 17/08/07 03:06:20 INFO Utils: Successfully started service 'sparkExecutorActorSystem' on port 39229. 17/08/07 03:06:20 INFO DiskBlockManager: Created local directory at /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1502070770869_0012/blockmgr-e6138329-b1c4-4f90-a113-dd05eba9730a 17/08/07 03:06:20 INFO MemoryStore: MemoryStore started with capacity 511.1 MB 17/08/07 03:06:20 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@xxx.xx.xx.xxx:53417 17/08/07 03:06:20 INFO CoarseGrainedExecutorBackend: Successfully registered with driver 17/08/07 03:06:20 INFO Executor: Starting executor ID 2 on host host4 17/08/07 03:06:20 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33596. 17/08/07 03:06:20 INFO NettyBlockTransferService: Server created on 33596 17/08/07 03:06:20 INFO BlockManagerMaster: Trying to register BlockManager 17/08/07 03:06:20 INFO BlockManagerMaster: Registered BlockManager 17/08/07 03:06:26 INFO CoarseGrainedExecutorBackend: Got assigned task 1 17/08/07 03:06:26 INFO Executor: Running task 0.0 in stage 1.0 (TID 1) 17/08/07 03:06:26 INFO TorrentBroadcast: Started reading broadcast variable 1 17/08/07 03:06:26 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 6.0 KB, free 511.1 MB) 17/08/07 03:06:26 INFO TorrentBroadcast: Reading broadcast variable 1 took 151 ms 17/08/07 03:06:26 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 14.4 KB, free 511.1 MB) 17/08/07 03:06:28 INFO GenerateUnsafeProjection: Code generated in 145.857734 ms 17/08/07 03:06:28 INFO RRDD: Times: boot = 0.792 s, init = 0.010 s, broadcast = 0.001 s, read-input = 0.000 s, compute = 0.002 s, write-output = 0.083 s, total = 0.888 s 17/08/07 03:06:28 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1374 bytes result sent to driver 17/08/07 03:06:28 INFO CoarseGrainedExecutorBackend: Got assigned task 2 17/08/07 03:06:28 INFO Executor: Running task 0.0 in stage 2.0 (TID 2) 17/08/07 03:06:28 INFO MapOutputTrackerWorker: Updating epoch to 1 and clearing cache 17/08/07 03:06:28 INFO TorrentBroadcast: Started reading broadcast variable 2 17/08/07 03:06:28 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1613.0 B, free 511.1 MB) 17/08/07 03:06:28 INFO TorrentBroadcast: Reading broadcast variable 2 took 17 ms 17/08/07 03:06:28 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.7 KB, free 511.1 MB) 17/08/07 03:06:28 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 0, fetching them 17/08/07 03:06:28 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@xxx.xx.xx.xxx:53417) 17/08/07 03:06:28 INFO MapOutputTrackerWorker: Got the output locations 17/08/07 03:06:28 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks 17/08/07 03:06:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 4 ms 17/08/07 03:06:28 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 1398 bytes result sent to driver 17/08/07 03:06:29 INFO CoarseGrainedExecutorBackend: Got assigned task 3 17/08/07 03:06:29 INFO Executor: Running task 0.0 in stage 3.0 (TID 3) 17/08/07 03:06:29 INFO TorrentBroadcast: Started reading broadcast variable 3 17/08/07 03:06:29 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 9.0 KB, free 511.1 MB) 17/08/07 03:06:29 INFO TorrentBroadcast: Reading broadcast variable 3 took 19 ms 17/08/07 03:06:29 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 20.7 KB, free 511.1 MB) 17/08/07 03:06:29 INFO GenerateUnsafeProjection: Code generated in 69.828337 ms 17/08/07 03:06:29 INFO GenerateSafeProjection: Code generated in 13.540783 ms 17/08/07 03:06:30 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3). 1374 bytes result sent to driver 17/08/07 03:06:30 INFO CoarseGrainedExecutorBackend: Got assigned task 4 17/08/07 03:06:30 INFO Executor: Running task 0.0 in stage 4.0 (TID 4) 17/08/07 03:06:30 INFO MapOutputTrackerWorker: Updating epoch to 2 and clearing cache 17/08/07 03:06:30 INFO TorrentBroadcast: Started reading broadcast variable 4 17/08/07 03:06:30 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 9.3 KB, free 511.1 MB) 17/08/07 03:06:30 INFO TorrentBroadcast: Reading broadcast variable 4 took 22 ms 17/08/07 03:06:30 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 21.2 KB, free 511.0 MB) 17/08/07 03:06:30 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 1, fetching them 17/08/07 03:06:30 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@xxx.xx.xx.xxx:53417) 17/08/07 03:06:30 INFO MapOutputTrackerWorker: Got the output locations 17/08/07 03:06:30 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks 17/08/07 03:06:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 17/08/07 03:06:30 INFO Executor: Finished task 0.0 in stage 4.0 (TID 4). 1085 bytes result sent to driver 17/08/07 03:06:32 INFO CoarseGrainedExecutorBackend: Got assigned task 6 17/08/07 03:06:32 INFO Executor: Running task 0.0 in stage 6.0 (TID 6) 17/08/07 03:06:32 INFO TorrentBroadcast: Started reading broadcast variable 6 17/08/07 03:06:32 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 10.1 KB, free 511.0 MB) 17/08/07 03:06:32 INFO TorrentBroadcast: Reading broadcast variable 6 took 19 ms 17/08/07 03:06:32 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 22.9 KB, free 511.0 MB) 17/08/07 03:06:32 INFO GenerateUnsafeProjection: Code generated in 38.415554 ms 17/08/07 03:06:32 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS 17/08/07 03:06:32 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS 17/08/07 03:06:32 INFO RRDD: Times: boot = 0.010 s, init = 0.005 s, broadcast = 0.000 s, read-input = 0.001 s, compute = 0.002 s, write-output = 0.073 s, total = 0.091 s 17/08/07 03:06:32 INFO Executor: Finished task 0.0 in stage 6.0 (TID 6). 1560 bytes result sent to driver 17/08/07 03:06:33 INFO CoarseGrainedExecutorBackend: Got assigned task 8 17/08/07 03:06:33 INFO Executor: Running task 0.0 in stage 8.0 (TID 8) 17/08/07 03:06:33 INFO TorrentBroadcast: Started reading broadcast variable 8 17/08/07 03:06:33 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 7.0 KB, free 511.1 MB) 17/08/07 03:06:33 INFO TorrentBroadcast: Reading broadcast variable 8 took 12 ms 17/08/07 03:06:33 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 16.9 KB, free 511.1 MB) 17/08/07 03:06:33 INFO GenerateUnsafeProjection: Code generated in 7.147903 ms 17/08/07 03:06:33 INFO GenerateMutableProjection: Code generated in 6.690512 ms 17/08/07 03:06:33 INFO GenerateMutableProjection: Code generated in 7.107527 ms 17/08/07 03:06:33 INFO GenerateUnsafeRowJoiner: Code generated in 6.006831 ms 17/08/07 03:06:33 INFO GenerateUnsafeProjection: Code generated in 6.628821 ms 17/08/07 03:06:33 INFO RRDD: Times: boot = 0.008 s, init = 0.003 s, broadcast = 0.000 s, read-input = 0.001 s, compute = 0.001 s, write-output = 0.070 s, total = 0.083 s 17/08/07 03:06:33 INFO Executor: Finished task 0.0 in stage 8.0 (TID 8). 1519 bytes result sent to driver 17/08/07 03:06:33 INFO CoarseGrainedExecutorBackend: Got assigned task 9 17/08/07 03:06:33 INFO Executor: Running task 0.0 in stage 9.0 (TID 9) 17/08/07 03:06:33 INFO MapOutputTrackerWorker: Updating epoch to 3 and clearing cache 17/08/07 03:06:33 INFO TorrentBroadcast: Started reading broadcast variable 9 17/08/07 03:06:33 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 7.4 KB, free 511.1 MB) 17/08/07 03:06:33 INFO TorrentBroadcast: Reading broadcast variable 9 took 12 ms 17/08/07 03:06:33 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 17.8 KB, free 511.0 MB) 17/08/07 03:06:33 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 2, fetching them 17/08/07 03:06:33 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@xxx.xx.xx.xxx:53417) 17/08/07 03:06:33 INFO MapOutputTrackerWorker: Got the output locations 17/08/07 03:06:33 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks 17/08/07 03:06:33 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 17/08/07 03:06:33 INFO GenerateMutableProjection: Code generated in 5.889433 ms 17/08/07 03:06:33 INFO GenerateMutableProjection: Code generated in 4.846414 ms 17/08/07 03:06:33 INFO Executor: Finished task 0.0 in stage 9.0 (TID 9). 1911 bytes result sent to driver 17/08/07 03:06:33 INFO CoarseGrainedExecutorBackend: Got assigned task 10 17/08/07 03:06:33 INFO Executor: Running task 0.0 in stage 10.0 (TID 10) 17/08/07 03:06:33 INFO TorrentBroadcast: Started reading broadcast variable 10 17/08/07 03:06:33 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 12.0 KB, free 511.0 MB) 17/08/07 03:06:33 INFO TorrentBroadcast: Reading broadcast variable 10 took 15 ms 17/08/07 03:06:33 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 27.2 KB, free 511.0 MB) 17/08/07 03:06:33 INFO GenerateUnsafeProjection: Code generated in 27.259051 ms 17/08/07 03:06:33 INFO GenerateMutableProjection: Code generated in 5.460526 ms 17/08/07 03:06:33 INFO GenerateMutableProjection: Code generated in 18.581499 ms 17/08/07 03:06:33 INFO GenerateUnsafeRowJoiner: Code generated in 4.470494 ms 17/08/07 03:06:33 INFO GenerateUnsafeProjection: Code generated in 4.768152 ms 17/08/07 03:06:34 INFO RRDD: Times: boot = 0.008 s, init = 0.004 s, broadcast = 0.000 s, read-input = 0.001 s, compute = 0.001 s, write-output = 0.068 s, total = 0.082 s 17/08/07 03:06:34 INFO Executor: Finished task 0.0 in stage 10.0 (TID 10). 1519 bytes result sent to driver 17/08/07 03:06:34 INFO CoarseGrainedExecutorBackend: Got assigned task 11 17/08/07 03:06:34 INFO Executor: Running task 0.0 in stage 11.0 (TID 11) 17/08/07 03:06:34 INFO MapOutputTrackerWorker: Updating epoch to 4 and clearing cache 17/08/07 03:06:34 INFO TorrentBroadcast: Started reading broadcast variable 11 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 12.4 KB, free 511.0 MB) 17/08/07 03:06:34 INFO TorrentBroadcast: Reading broadcast variable 11 took 13 ms 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 28.2 KB, free 511.0 MB) 17/08/07 03:06:34 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 3, fetching them 17/08/07 03:06:34 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@xxx.xx.xx.xxx:53417) 17/08/07 03:06:34 INFO MapOutputTrackerWorker: Got the output locations 17/08/07 03:06:34 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks 17/08/07 03:06:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 17/08/07 03:06:34 INFO GenerateMutableProjection: Code generated in 8.266109 ms 17/08/07 03:06:34 INFO GenerateMutableProjection: Code generated in 3.835426 ms 17/08/07 03:06:34 INFO GenerateSafeProjection: Code generated in 3.489347 ms 17/08/07 03:06:34 INFO Executor: Finished task 0.0 in stage 11.0 (TID 11). 1991 bytes result sent to driver 17/08/07 03:06:34 INFO CoarseGrainedExecutorBackend: Got assigned task 12 17/08/07 03:06:34 INFO Executor: Running task 0.0 in stage 12.0 (TID 12) 17/08/07 03:06:34 INFO TorrentBroadcast: Started reading broadcast variable 12 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 10.2 KB, free 511.0 MB) 17/08/07 03:06:34 INFO TorrentBroadcast: Reading broadcast variable 12 took 15 ms 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 22.8 KB, free 510.9 MB) 17/08/07 03:06:34 INFO GenerateUnsafeProjection: Code generated in 22.165839 ms 17/08/07 03:06:34 INFO Executor: Finished task 0.0 in stage 12.0 (TID 12). 1644 bytes result sent to driver 17/08/07 03:06:34 INFO CoarseGrainedExecutorBackend: Got assigned task 15 17/08/07 03:06:34 INFO Executor: Running task 0.0 in stage 15.0 (TID 15) 17/08/07 03:06:34 INFO MapOutputTrackerWorker: Updating epoch to 5 and clearing cache 17/08/07 03:06:34 INFO TorrentBroadcast: Started reading broadcast variable 15 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.9 MB) 17/08/07 03:06:34 INFO TorrentBroadcast: Reading broadcast variable 15 took 15 ms 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_15 stored as values in memory (estimated size 21.0 KB, free 510.9 MB) 17/08/07 03:06:34 INFO CacheManager: Partition rdd_52_0 not found, computing it 17/08/07 03:06:34 INFO GeneratePredicate: Code generated in 4.854431 ms 17/08/07 03:06:34 INFO GenerateUnsafeProjection: Code generated in 17.619355 ms 17/08/07 03:06:34 INFO RRDD: Times: boot = 0.007 s, init = 0.003 s, broadcast = 0.000 s, read-input = 0.001 s, compute = 0.002 s, write-output = 0.068 s, total = 0.081 s 17/08/07 03:06:34 INFO MemoryStore: Block rdd_52_0 stored as values in memory (estimated size 8.2 KB, free 510.9 MB) 17/08/07 03:06:34 INFO Executor: Finished task 0.0 in stage 15.0 (TID 15). 2907 bytes result sent to driver 17/08/07 03:06:35 INFO CoarseGrainedExecutorBackend: Got assigned task 16 17/08/07 03:06:35 INFO Executor: Running task 0.0 in stage 16.0 (TID 16) 17/08/07 03:06:35 INFO TorrentBroadcast: Started reading broadcast variable 16 17/08/07 03:06:35 INFO MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.9 MB) 17/08/07 03:06:35 INFO TorrentBroadcast: Reading broadcast variable 16 took 14 ms 17/08/07 03:06:35 INFO MemoryStore: Block broadcast_16 stored as values in memory (estimated size 21.0 KB, free 510.9 MB) 17/08/07 03:06:35 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:35 INFO Executor: Finished task 0.0 in stage 16.0 (TID 16). 2898 bytes result sent to driver 17/08/07 03:06:35 INFO CoarseGrainedExecutorBackend: Got assigned task 17 17/08/07 03:06:35 INFO Executor: Running task 0.0 in stage 17.0 (TID 17) 17/08/07 03:06:35 INFO TorrentBroadcast: Started reading broadcast variable 17 17/08/07 03:06:35 INFO MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.9 MB) 17/08/07 03:06:35 INFO TorrentBroadcast: Reading broadcast variable 17 took 14 ms 17/08/07 03:06:35 INFO MemoryStore: Block broadcast_17 stored as values in memory (estimated size 21.0 KB, free 510.8 MB) 17/08/07 03:06:35 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:35 INFO Executor: Finished task 0.0 in stage 17.0 (TID 17). 2898 bytes result sent to driver 17/08/07 03:06:35 INFO CoarseGrainedExecutorBackend: Got assigned task 18 17/08/07 03:06:35 INFO Executor: Running task 0.0 in stage 18.0 (TID 18) 17/08/07 03:06:35 INFO TorrentBroadcast: Started reading broadcast variable 18 17/08/07 03:06:35 INFO MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.8 MB) 17/08/07 03:06:35 INFO TorrentBroadcast: Reading broadcast variable 18 took 14 ms 17/08/07 03:06:35 INFO MemoryStore: Block broadcast_18 stored as values in memory (estimated size 21.0 KB, free 510.8 MB) 17/08/07 03:06:35 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:35 INFO Executor: Finished task 0.0 in stage 18.0 (TID 18). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 19 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 19.0 (TID 19) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 19 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.8 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 19 took 12 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_19 stored as values in memory (estimated size 21.0 KB, free 510.8 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 19.0 (TID 19). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 20 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 20.0 (TID 20) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 20 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.8 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 20 took 15 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_20 stored as values in memory (estimated size 21.0 KB, free 510.8 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 20.0 (TID 20). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 21 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 21.0 (TID 21) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 21 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.7 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 21 took 16 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_21 stored as values in memory (estimated size 21.0 KB, free 510.7 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 21.0 (TID 21). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 22 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 22.0 (TID 22) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 22 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_22_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.7 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 22 took 17 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_22 stored as values in memory (estimated size 21.0 KB, free 510.7 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 22.0 (TID 22). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 23 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 23.0 (TID 23) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 23 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_23_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.7 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 23 took 14 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_23 stored as values in memory (estimated size 21.0 KB, free 510.7 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 23.0 (TID 23). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 24 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 24.0 (TID 24) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 24 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_24_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.7 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 24 took 13 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_24 stored as values in memory (estimated size 21.0 KB, free 510.6 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 24.0 (TID 24). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 25 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 25.0 (TID 25) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 25 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_25_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.6 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 25 took 14 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_25 stored as values in memory (estimated size 21.0 KB, free 510.6 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 25.0 (TID 25). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 26 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 26.0 (TID 26) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 26 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_26_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.6 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 26 took 14 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_26 stored as values in memory (estimated size 21.0 KB, free 510.6 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 26.0 (TID 26). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 27 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 27.0 (TID 27) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 27 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_27_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.6 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 27 took 14 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_27 stored as values in memory (estimated size 21.0 KB, free 510.5 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 27.0 (TID 27). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 28 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 28.0 (TID 28) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 28 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_28_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.6 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 28 took 11 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_28 stored as values in memory (estimated size 21.0 KB, free 510.6 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 28.0 (TID 28). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 29 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 29.0 (TID 29) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 29 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_29_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.8 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 29 took 15 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_29 stored as values in memory (estimated size 21.0 KB, free 510.8 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 29.0 (TID 29). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 30 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 30.0 (TID 30) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 30 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_30_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.9 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 30 took 13 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_30 stored as values in memory (estimated size 21.0 KB, free 510.9 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 30.0 (TID 30). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 31 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 31.0 (TID 31) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 31 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_31_piece0 stored as bytes in memory (estimated size 9.0 KB, free 511.0 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 31 took 11 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_31 stored as values in memory (estimated size 21.0 KB, free 511.0 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 31.0 (TID 31). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 32 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 32.0 (TID 32) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 32 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_32_piece0 stored as bytes in memory (estimated size 9.0 KB, free 511.0 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 32 took 11 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_32 stored as values in memory (estimated size 21.0 KB, free 511.0 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 32.0 (TID 32). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 33 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 33.0 (TID 33) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 33 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_33_piece0 stored as bytes in memory (estimated size 9.0 KB, free 511.0 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 33 took 10 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_33 stored as values in memory (estimated size 21.0 KB, free 510.9 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 33.0 (TID 33). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 34 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 34.0 (TID 34) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 34 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_34_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.9 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 34 took 10 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_34 stored as values in memory (estimated size 21.0 KB, free 510.9 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 34.0 (TID 34). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 35 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 35.0 (TID 35) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 35 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_35_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.9 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 35 took 10 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_35 stored as values in memory (estimated size 21.0 KB, free 510.9 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 35.0 (TID 35). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 36 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 36.0 (TID 36) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 36 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_36_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.9 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 36 took 9 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_36 stored as values in memory (estimated size 21.0 KB, free 510.9 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 36.0 (TID 36). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 37 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 37.0 (TID 37) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 37 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_37_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.8 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 37 took 9 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_37 stored as values in memory (estimated size 21.0 KB, free 510.8 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 37.0 (TID 37). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 38 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 38.0 (TID 38) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 38 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_38_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.8 MB) 17/08/07 03:06:36 INFO TorrentBroadcast: Reading broadcast variable 38 took 10 ms 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_38 stored as values in memory (estimated size 21.0 KB, free 510.8 MB) 17/08/07 03:06:36 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:36 INFO Executor: Finished task 0.0 in stage 38.0 (TID 38). 2898 bytes result sent to driver 17/08/07 03:06:36 INFO CoarseGrainedExecutorBackend: Got assigned task 39 17/08/07 03:06:36 INFO Executor: Running task 0.0 in stage 39.0 (TID 39) 17/08/07 03:06:36 INFO TorrentBroadcast: Started reading broadcast variable 39 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_39_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.8 MB) 17/08/07 03:06:37 INFO TorrentBroadcast: Reading broadcast variable 39 took 9 ms 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_39 stored as values in memory (estimated size 21.0 KB, free 510.8 MB) 17/08/07 03:06:37 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:37 INFO Executor: Finished task 0.0 in stage 39.0 (TID 39). 2898 bytes result sent to driver 17/08/07 03:06:37 INFO CoarseGrainedExecutorBackend: Got assigned task 40 17/08/07 03:06:37 INFO Executor: Running task 0.0 in stage 40.0 (TID 40) 17/08/07 03:06:37 INFO TorrentBroadcast: Started reading broadcast variable 40 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_40_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.8 MB) 17/08/07 03:06:37 INFO TorrentBroadcast: Reading broadcast variable 40 took 9 ms 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_40 stored as values in memory (estimated size 21.0 KB, free 510.7 MB) 17/08/07 03:06:37 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:37 INFO Executor: Finished task 0.0 in stage 40.0 (TID 40). 2898 bytes result sent to driver 17/08/07 03:06:37 INFO CoarseGrainedExecutorBackend: Got assigned task 41 17/08/07 03:06:37 INFO Executor: Running task 0.0 in stage 41.0 (TID 41) 17/08/07 03:06:37 INFO TorrentBroadcast: Started reading broadcast variable 41 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_41_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.7 MB) 17/08/07 03:06:37 INFO TorrentBroadcast: Reading broadcast variable 41 took 9 ms 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_41 stored as values in memory (estimated size 21.0 KB, free 510.7 MB) 17/08/07 03:06:37 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:37 INFO Executor: Finished task 0.0 in stage 41.0 (TID 41). 2898 bytes result sent to driver 17/08/07 03:06:37 INFO CoarseGrainedExecutorBackend: Got assigned task 42 17/08/07 03:06:37 INFO Executor: Running task 0.0 in stage 42.0 (TID 42) 17/08/07 03:06:37 INFO TorrentBroadcast: Started reading broadcast variable 42 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_42_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.7 MB) 17/08/07 03:06:37 INFO TorrentBroadcast: Reading broadcast variable 42 took 9 ms 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_42 stored as values in memory (estimated size 21.0 KB, free 510.7 MB) 17/08/07 03:06:37 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:37 INFO Executor: Finished task 0.0 in stage 42.0 (TID 42). 2898 bytes result sent to driver 17/08/07 03:06:37 INFO CoarseGrainedExecutorBackend: Got assigned task 43 17/08/07 03:06:37 INFO Executor: Running task 0.0 in stage 43.0 (TID 43) 17/08/07 03:06:37 INFO TorrentBroadcast: Started reading broadcast variable 43 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_43_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.7 MB) 17/08/07 03:06:37 INFO TorrentBroadcast: Reading broadcast variable 43 took 9 ms 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_43 stored as values in memory (estimated size 21.0 KB, free 510.6 MB) 17/08/07 03:06:37 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:37 INFO Executor: Finished task 0.0 in stage 43.0 (TID 43). 2898 bytes result sent to driver 17/08/07 03:06:37 INFO CoarseGrainedExecutorBackend: Got assigned task 44 17/08/07 03:06:37 INFO Executor: Running task 0.0 in stage 44.0 (TID 44) 17/08/07 03:06:37 INFO TorrentBroadcast: Started reading broadcast variable 44 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_44_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.6 MB) 17/08/07 03:06:37 INFO TorrentBroadcast: Reading broadcast variable 44 took 11 ms 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_44 stored as values in memory (estimated size 21.0 KB, free 510.6 MB) 17/08/07 03:06:37 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:37 INFO Executor: Finished task 0.0 in stage 44.0 (TID 44). 2898 bytes result sent to driver 17/08/07 03:06:37 INFO CoarseGrainedExecutorBackend: Got assigned task 45 17/08/07 03:06:37 INFO Executor: Running task 0.0 in stage 45.0 (TID 45) 17/08/07 03:06:37 INFO TorrentBroadcast: Started reading broadcast variable 45 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_45_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.6 MB) 17/08/07 03:06:37 INFO TorrentBroadcast: Reading broadcast variable 45 took 11 ms 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_45 stored as values in memory (estimated size 21.0 KB, free 510.6 MB) 17/08/07 03:06:37 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:37 INFO Executor: Finished task 0.0 in stage 45.0 (TID 45). 2898 bytes result sent to driver 17/08/07 03:06:37 INFO CoarseGrainedExecutorBackend: Got assigned task 46 17/08/07 03:06:37 INFO Executor: Running task 0.0 in stage 46.0 (TID 46) 17/08/07 03:06:37 INFO TorrentBroadcast: Started reading broadcast variable 46 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_46_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.6 MB) 17/08/07 03:06:37 INFO TorrentBroadcast: Reading broadcast variable 46 took 7 ms 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_46 stored as values in memory (estimated size 21.0 KB, free 510.6 MB) 17/08/07 03:06:37 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:37 INFO Executor: Finished task 0.0 in stage 46.0 (TID 46). 2898 bytes result sent to driver 17/08/07 03:06:37 INFO CoarseGrainedExecutorBackend: Got assigned task 47 17/08/07 03:06:37 INFO Executor: Running task 0.0 in stage 47.0 (TID 47) 17/08/07 03:06:37 INFO TorrentBroadcast: Started reading broadcast variable 47 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_47_piece0 stored as bytes in memory (estimated size 9.0 KB, free 510.6 MB) 17/08/07 03:06:37 INFO TorrentBroadcast: Reading broadcast variable 47 took 7 ms 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_47 stored as values in memory (estimated size 21.0 KB, free 510.5 MB) 17/08/07 03:06:37 INFO BlockManager: Found block rdd_52_0 locally 17/08/07 03:06:37 INFO Executor: Finished task 0.0 in stage 47.0 (TID 47). 2898 bytes result sent to driver 17/08/07 03:06:37 INFO BlockManager: Removing RDD 52 17/08/07 03:06:37 INFO BlockManager: Removing RDD 52 17/08/07 03:06:37 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown 17/08/07 03:06:37 INFO MemoryStore: MemoryStore cleared 17/08/07 03:06:37 INFO BlockManager: BlockManager stopped 17/08/07 03:06:37 INFO CoarseGrainedExecutorBackend: Driver from host4:53417 disconnected during shutdown 17/08/07 03:06:37 INFO CoarseGrainedExecutorBackend: Driver from xxx.xx.xx.xxx:53417 disconnected during shutdown 17/08/07 03:06:37 INFO ShutdownHookManager: Shutdown hook called End of LogType:stderr LogType:stdout Log Upload Time:Mon Aug 07 03:06:40 +0000 2017 LogLength:0 Log Contents: End of LogType:stdout Container: container_e02_1502070770869_0012_01_000001 on host4_25454_1502075200681 LogAggregationType: AGGREGATED ============================================================================================================================ LogType:directory.info Log Upload Time:Mon Aug 07 03:06:40 +0000 2017 LogLength:12201 Log Contents: ls -l: total 12 -rw------- 1 hrt_qa hadoop 520 Aug 7 03:06 container_tokens -rwx------ 1 hrt_qa hadoop 6498 Aug 7 03:06 launch_container.sh lrwxrwxrwx 1 hrt_qa hadoop 60 Aug 7 03:06 ml.R -> /grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/15/ml.R lrwxrwxrwx 1 hrt_qa hadoop 93 Aug 7 03:06 __spark_conf__ -> /grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/16/__spark_conf__5875502619868816278.zip lrwxrwxrwx 1 hrt_qa hadoop 61 Aug 7 03:06 __spark__.jar -> /grid/0/hadoop/yarn/local/filecache/12/spark-hdp-assembly.jar lrwxrwxrwx 1 hrt_qa hadoop 66 Aug 7 03:06 sparkr -> /grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/14/sparkr.zip drwxr-s--- 2 hrt_qa hadoop 6 Aug 7 03:06 tmp find -L . -maxdepth 5 -ls: 1747998 0 drwxr-s--- 3 hrt_qa hadoop 130 Aug 7 03:06 . 144779289 0 drwxr-s--- 2 hrt_qa hadoop 6 Aug 7 03:06 ./tmp 1747999 8 -rwx------ 1 hrt_qa hadoop 6498 Aug 7 03:06 ./launch_container.sh 1748000 4 -rw------- 1 hrt_qa hadoop 520 Aug 7 03:06 ./container_tokens 1010802710 0 drwx------ 3 hrt_qa hrt_qa 19 Aug 7 03:06 ./sparkr 1747995 0 drwx------ 10 hrt_qa hrt_qa 139 Aug 7 03:06 ./sparkr/SparkR 1747996 8 -r-x------ 1 hrt_qa hrt_qa 7018 Aug 7 03:06 ./sparkr/SparkR/NAMESPACE 70865960 0 drwx------ 2 hrt_qa hrt_qa 88 Aug 7 03:06 ./sparkr/SparkR/Meta 70865961 4 -r-x------ 1 hrt_qa hrt_qa 164 Aug 7 03:06 ./sparkr/SparkR/Meta/Rd.rds 70865962 4 -r-x------ 1 hrt_qa hrt_qa 188 Aug 7 03:06 ./sparkr/SparkR/Meta/hsearch.rds 70865963 4 -r-x------ 1 hrt_qa hrt_qa 1717 Aug 7 03:06 ./sparkr/SparkR/Meta/nsInfo.rds 70865964 4 -r-x------ 1 hrt_qa hrt_qa 746 Aug 7 03:06 ./sparkr/SparkR/Meta/package.rds 70865965 4 -r-x------ 1 hrt_qa hrt_qa 38 Aug 7 03:06 ./sparkr/SparkR/Meta/links.rds 144779287 0 drwx------ 2 hrt_qa hrt_qa 38 Aug 7 03:06 ./sparkr/SparkR/test_support 144779288 4 -r-x------ 1 hrt_qa hrt_qa 2886 Aug 7 03:06 ./sparkr/SparkR/test_support/sparktestjar_2.10-1.0.jar 205317228 0 drwx------ 2 hrt_qa hrt_qa 88 Aug 7 03:06 ./sparkr/SparkR/help 205317229 4 -r-x------ 1 hrt_qa hrt_qa 64 Aug 7 03:06 ./sparkr/SparkR/help/paths.rds 205317230 0 -r-x------ 1 hrt_qa hrt_qa 0 Aug 7 03:06 ./sparkr/SparkR/help/SparkR.rdb 205317231 4 -r-x------ 1 hrt_qa hrt_qa 60 Aug 7 03:06 ./sparkr/SparkR/help/aliases.rds 205317232 0 -r-x------ 1 hrt_qa hrt_qa 0 Aug 7 03:06 ./sparkr/SparkR/help/AnIndex 205317233 4 -r-x------ 1 hrt_qa hrt_qa 113 Aug 7 03:06 ./sparkr/SparkR/help/SparkR.rdx 271809573 0 drwx------ 3 hrt_qa hrt_qa 21 Aug 7 03:06 ./sparkr/SparkR/tests 340968465 4 drwx------ 2 hrt_qa hrt_qa 4096 Aug 7 03:06 ./sparkr/SparkR/tests/testthat 340968505 28 -r-x------ 1 hrt_qa hrt_qa 27490 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_rdd.R 340968506 4 -r-x------ 1 hrt_qa hrt_qa 1957 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_client.R 340968507 4 -r-x------ 1 hrt_qa hrt_qa 3981 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_context.R 340968508 4 -r-x------ 1 hrt_qa hrt_qa 3691 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_binary_function.R 340968509 4 -r-x------ 1 hrt_qa hrt_qa 3171 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_take.R 340968510 8 -r-x------ 1 hrt_qa hrt_qa 5137 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_textFile.R 340968511 8 -r-x------ 1 hrt_qa hrt_qa 4380 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_parallelize_collect.R 340967452 8 -r-x------ 1 hrt_qa hrt_qa 4859 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_mllib.R 340912155 4 -r-x------ 1 hrt_qa hrt_qa 1241 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/jarTest.R 340912161 4 -r-x------ 1 hrt_qa hrt_qa 1542 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_includeJAR.R 340912162 4 -r-x------ 1 hrt_qa hrt_qa 3064 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_binaryFile.R 340430848 4 -r-x------ 1 hrt_qa hrt_qa 1611 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_broadcast.R 340430851 4 -r-x------ 1 hrt_qa hrt_qa 1810 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_includePackage.R 340432896 4 -r-x------ 1 hrt_qa hrt_qa 2327 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_Serde.R 340432897 4 -r-x------ 1 hrt_qa hrt_qa 996 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/packageInAJarTest.R 340432898 8 -r-x------ 1 hrt_qa hrt_qa 5201 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_utils.R 340432899 8 -r-x------ 1 hrt_qa hrt_qa 7922 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_shuffle.R 340432900 72 -r-x------ 1 hrt_qa hrt_qa 70877 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_sparkSQL.R 1747997 4 -r-x------ 1 hrt_qa hrt_qa 678 Aug 7 03:06 ./sparkr/SparkR/DESCRIPTION 403415064 0 drwx------ 2 hrt_qa hrt_qa 37 Aug 7 03:06 ./sparkr/SparkR/html 403415065 4 -r-x------ 1 hrt_qa hrt_qa 923 Aug 7 03:06 ./sparkr/SparkR/html/00Index.html 403415066 4 -r-x------ 1 hrt_qa hrt_qa 1319 Aug 7 03:06 ./sparkr/SparkR/html/R.css 472623140 0 drwx------ 2 hrt_qa hrt_qa 53 Aug 7 03:06 ./sparkr/SparkR/R 472623141 4 -r-x------ 1 hrt_qa hrt_qa 1056 Aug 7 03:06 ./sparkr/SparkR/R/SparkR 472623142 804 -r-x------ 1 hrt_qa hrt_qa 821006 Aug 7 03:06 ./sparkr/SparkR/R/SparkR.rdb 472653849 20 -r-x------ 1 hrt_qa hrt_qa 19182 Aug 7 03:06 ./sparkr/SparkR/R/SparkR.rdx 540117073 0 drwx------ 2 hrt_qa hrt_qa 36 Aug 7 03:06 ./sparkr/SparkR/profile 540117074 4 -r-x------ 1 hrt_qa hrt_qa 1696 Aug 7 03:06 ./sparkr/SparkR/profile/shell.R 540117075 4 -r-x------ 1 hrt_qa hrt_qa 958 Aug 7 03:06 ./sparkr/SparkR/profile/general.R 608539664 0 drwx------ 2 hrt_qa hrt_qa 36 Aug 7 03:06 ./sparkr/SparkR/worker 608539665 8 -r-x------ 1 hrt_qa hrt_qa 5884 Aug 7 03:06 ./sparkr/SparkR/worker/worker.R 608539666 4 -r-x------ 1 hrt_qa hrt_qa 1863 Aug 7 03:06 ./sparkr/SparkR/worker/daemon.R 875621379 186408 -r-xr-xr-x 1 yarn hadoop 190878097 Aug 7 02:35 ./__spark__.jar 876876813 4 drwx------ 2 hrt_qa hrt_qa 4096 Aug 7 03:06 ./__spark_conf__ 876876814 4 -r-x------ 1 hrt_qa hrt_qa 620 Aug 7 03:06 ./__spark_conf__/log4j.properties 876876815 8 -r-x------ 1 hrt_qa hrt_qa 6606 Aug 7 03:06 ./__spark_conf__/hadoop-env.sh 876876816 8 -r-x------ 1 hrt_qa hrt_qa 7831 Aug 7 03:06 ./__spark_conf__/mapred-site.xml 876876817 4 -r-x------ 1 hrt_qa hrt_qa 3085 Aug 7 03:06 ./__spark_conf__/hadoop-metrics2.properties 876876818 28 -r-x------ 1 hrt_qa hrt_qa 24749 Aug 7 03:06 ./__spark_conf__/yarn-site.xml 876876819 4 -r-x------ 1 hrt_qa hrt_qa 3979 Aug 7 03:06 ./__spark_conf__/hadoop-env.cmd 876876820 8 -r-x------ 1 hrt_qa hrt_qa 7596 Aug 7 03:06 ./__spark_conf__/core-site.xml 876876821 0 -r-x------ 1 hrt_qa hrt_qa 0 Aug 7 03:06 ./__spark_conf__/yarn.exclude 876876822 4 -r-x------ 1 hrt_qa hrt_qa 1 Aug 7 03:06 ./__spark_conf__/dfs.exclude 876876823 4 -r-x------ 1 hrt_qa hrt_qa 1631 Aug 7 03:06 ./__spark_conf__/kms-log4j.properties 876876824 4 -r-x------ 1 hrt_qa hrt_qa 2250 Aug 7 03:06 ./__spark_conf__/yarn-env.cmd 876876825 4 -r-x------ 1 hrt_qa hrt_qa 917 Aug 7 03:06 ./__spark_conf__/ssl-client.xml 876876826 4 -r-x------ 1 hrt_qa hrt_qa 2313 Aug 7 03:06 ./__spark_conf__/capacity-scheduler.xml 876876827 4 -r-x------ 1 hrt_qa hrt_qa 2610 Aug 7 03:06 ./__spark_conf__/ranger-hdfs-audit.xml 876876828 4 -r-x------ 1 hrt_qa hrt_qa 3518 Aug 7 03:06 ./__spark_conf__/kms-acls.xml 876876829 4 -r-x------ 1 hrt_qa hrt_qa 758 Aug 7 03:06 ./__spark_conf__/mapred-site.xml.template 876876830 4 -r-x------ 1 hrt_qa hrt_qa 2358 Aug 7 03:06 ./__spark_conf__/topology_script.py 876876831 4 -r-x------ 1 hrt_qa hrt_qa 333 Aug 7 03:06 ./__spark_conf__/yarn_nm_jaas.conf 876876832 4 -r-x------ 1 hrt_qa hrt_qa 1335 Aug 7 03:06 ./__spark_conf__/configuration.xsl 876876833 4 -r-x------ 1 hrt_qa hrt_qa 64 Aug 7 03:06 ./__spark_conf__/ranger-security.xml 876876834 8 -r-x------ 1 hrt_qa hrt_qa 5532 Aug 7 03:06 ./__spark_conf__/yarn-env.sh 876876835 12 -r-x------ 1 hrt_qa hrt_qa 11405 Aug 7 03:06 ./__spark_conf__/hdfs-site.xml 876876836 4 -r-x------ 1 hrt_qa hrt_qa 337 Aug 7 03:06 ./__spark_conf__/yarn_ats_jaas.conf 876876837 4 -r-x------ 1 hrt_qa hrt_qa 1020 Aug 7 03:06 ./__spark_conf__/commons-logging.properties 876876838 4 -r-x------ 1 hrt_qa hrt_qa 333 Aug 7 03:06 ./__spark_conf__/hdfs_nn_jaas.conf 876876839 4 -r-x------ 1 hrt_qa hrt_qa 333 Aug 7 03:06 ./__spark_conf__/hdfs_dn_jaas.conf 876876840 4 -r-x------ 1 hrt_qa hrt_qa 1033 Aug 7 03:06 ./__spark_conf__/container-executor.cfg 876876841 4 -r-x------ 1 hrt_qa hrt_qa 571 Aug 7 03:06 ./__spark_conf__/yarn_jaas.conf 876876842 8 -r-x------ 1 hrt_qa hrt_qa 4221 Aug 7 03:06 ./__spark_conf__/task-log4j.properties 876876843 4 -r-x------ 1 hrt_qa hrt_qa 2490 Aug 7 03:06 ./__spark_conf__/hadoop-metrics.properties 876876844 4 -r-x------ 1 hrt_qa hrt_qa 984 Aug 7 03:06 ./__spark_conf__/mapred-env.sh 876876845 4 -r-x------ 1 hrt_qa hrt_qa 1602 Aug 7 03:06 ./__spark_conf__/health_check 876876846 4 -r-x------ 1 hrt_qa hrt_qa 1001 Aug 7 03:06 ./__spark_conf__/ranger-policymgr-ssl-yarn.xml 876876847 4 -r-x------ 1 hrt_qa hrt_qa 320 Aug 7 03:06 ./__spark_conf__/mapred_jaas.conf 876876848 4 -r-x------ 1 hrt_qa hrt_qa 2316 Aug 7 03:06 ./__spark_conf__/ssl-client.xml.example 876876849 4 -r-x------ 1 hrt_qa hrt_qa 1527 Aug 7 03:06 ./__spark_conf__/kms-env.sh 876876850 4 -r-x------ 1 hrt_qa hrt_qa 1308 Aug 7 03:06 ./__spark_conf__/hadoop-policy.xml 876876851 4 -r-x------ 1 hrt_qa hrt_qa 2610 Aug 7 03:06 ./__spark_conf__/ranger-yarn-audit.xml 876876852 4 -r-x------ 1 hrt_qa hrt_qa 145 Aug 7 03:06 ./__spark_conf__/slaves 876876853 4 -r-x------ 1 hrt_qa hrt_qa 287 Aug 7 03:06 ./__spark_conf__/topology_mappings.data 876876854 4 -r-x------ 1 hrt_qa hrt_qa 1049 Aug 7 03:06 ./__spark_conf__/ranger-hdfs-security.xml 876876855 8 -r-x------ 1 hrt_qa hrt_qa 4956 Aug 7 03:06 ./__spark_conf__/metrics.properties 876876856 4 -r-x------ 1 hrt_qa hrt_qa 951 Aug 7 03:06 ./__spark_conf__/mapred-env.cmd 876876857 4 -r-x------ 1 hrt_qa hrt_qa 1054 Aug 7 03:06 ./__spark_conf__/ssl-server.xml 876876858 4 -r-x------ 1 hrt_qa hrt_qa 333 Aug 7 03:06 ./__spark_conf__/hdfs_jn_jaas.conf 876876859 4 -r-x------ 1 hrt_qa hrt_qa 2697 Aug 7 03:06 ./__spark_conf__/ssl-server.xml.example 876876860 4 -r-x------ 1 hrt_qa hrt_qa 995 Aug 7 03:06 ./__spark_conf__/ranger-policymgr-ssl.xml 876876861 4 -r-x------ 1 hrt_qa hrt_qa 935 Aug 7 03:06 ./__spark_conf__/ranger-yarn-security.xml 876876862 4 -r-x------ 1 hrt_qa hrt_qa 946 Aug 7 03:06 ./__spark_conf__/taskcontroller.cfg 876876863 8 -r-x------ 1 hrt_qa hrt_qa 4113 Aug 7 03:06 ./__spark_conf__/mapred-queues.xml.template 875621406 8 -r-x------ 1 hrt_qa hrt_qa 5511 Aug 7 03:06 ./__spark_conf__/kms-site.xml 875621407 4 -r-x------ 1 hrt_qa hrt_qa 1661 Aug 7 03:06 ./__spark_conf__/__spark_conf__.properties 676291710 4 -r-x------ 1 hrt_qa hrt_qa 1903 Aug 7 03:06 ./ml.R broken symlinks(find -L . -maxdepth 5 -type l -ls): End of LogType:directory.info LogType:launch_container.sh Log Upload Time:Mon Aug 07 03:06:40 +0000 2017 LogLength:6498 Log Contents: #!/bin/bash export SPARK_YARN_STAGING_DIR=".sparkStaging/application_1502070770869_0012" export PATH="/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/var/lib/ambari-agent" export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/hdp/current/hadoop-client/conf"} export MAX_APP_ATTEMPTS="20" export JAVA_HOME=${JAVA_HOME:-"/usr/lib/jvm/java-openjdk"} export SPARK_YARN_CACHE_FILES="hdfs://mycluster/hdp/apps/2.6.2.0-154/spark/spark-hdp-assembly.jar#__spark__.jar,hdfs://mycluster/user/hrt_qa/.sparkStaging/application_1502070770869_0012/ml.R#ml.R" export LANG="en_US.UTF-8" export APP_SUBMIT_TIME_ENV="1502075166505" export NM_HOST="host4" export SPARK_YARN_CACHE_FILES_FILE_SIZES="190878097,1903" export SPARK_YARN_CACHE_ARCHIVES_TIME_STAMPS="1502075165966,1502075166070" export LOGNAME="hrt_qa" export JVM_PID="$$" export PWD="/grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1502070770869_0012/container_e02_1502070770869_0012_01_000001" export LOCAL_DIRS="/grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1502070770869_0012" export APPLICATION_WEB_PROXY_BASE="/proxy/application_1502070770869_0012" export NM_HTTP_PORT="54321" export LOG_DIRS="/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000001" export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= " export NM_PORT="25454" export SPARK_YARN_CACHE_FILES_TIME_STAMPS="1502064724925,1502075165821" export USER="hrt_qa" export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/hdp/current/hadoop-yarn-nodemanager"} export SPARK_YARN_CACHE_ARCHIVES="hdfs://mycluster/user/hrt_qa/.sparkStaging/application_1502070770869_0012/sparkr.zip#sparkr,hdfs://mycluster/user/hrt_qa/.sparkStaging/application_1502070770869_0012/__spark_conf__5875502619868816278.zip#__spark_conf__" export CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark__.jar:/etc/hadoop/conf:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:/usr/hdp/current/ext/hadoop/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/2.6.2.0-154/hadoop/lib/hadoop-lzo-0.6.0.2.6.2.0-154.jar:/etc/hadoop/conf/secure:/usr/hdp/current/ext/hadoop/*" export SPARK_YARN_CACHE_ARCHIVES_FILE_SIZES="555499,144034" export SPARK_YARN_MODE="true" export SPARK_YARN_CACHE_FILES_VISIBILITIES="PUBLIC,PRIVATE" export HADOOP_TOKEN_FILE_LOCATION="/grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1502070770869_0012/container_e02_1502070770869_0012_01_000001/container_tokens" export NM_AUX_SERVICE_spark_shuffle="" export SPARK_USER="hrt_qa" export LOCAL_USER_DIRS="/grid/0/hadoop/yarn/local/usercache/hrt_qa/" export HADOOP_HOME="/usr/hdp/2.6.2.0-154/hadoop" export SPARK_YARN_CACHE_ARCHIVES_VISIBILITIES="PRIVATE,PRIVATE" export HOME="/home/" export NM_AUX_SERVICE_spark2_shuffle="" export CONTAINER_ID="container_e02_1502070770869_0012_01_000001" export MALLOC_ARENA_MAX="4" ln -sf "/grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/14/sparkr.zip" "sparkr" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/grid/0/hadoop/yarn/local/filecache/12/spark-hdp-assembly.jar" "__spark__.jar" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/16/__spark_conf__5875502619868816278.zip" "__spark_conf__" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/15/ml.R" "ml.R" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi # Creating copy of launch script cp "launch_container.sh" "/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000001/launch_container.sh" chmod 640 "/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000001/launch_container.sh" # Determining directory contents echo "ls -l:" 1>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000001/directory.info" ls -l 1>>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000001/directory.info" echo "find -L . -maxdepth 5 -ls:" 1>>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000001/directory.info" find -L . -maxdepth 5 -ls 1>>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000001/directory.info" echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000001/directory.info" find -L . -maxdepth 5 -type l -ls 1>>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000001/directory.info" exec /bin/bash -c "LD_LIBRARY_PATH="/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dhdp.version=2.6.2.0-154 -Dspark.yarn.app.container.log.dir=/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.deploy.RRunner' --primary-r-file ml.R --arg 'ml.R' --executor-memory 1024m --executor-cores 1 --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000001/stdout 2> /grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000001/stderr" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi End of LogType:launch_container.sh LogType:stderr Log Upload Time:Mon Aug 07 03:06:40 +0000 2017 LogLength:153513 Log Contents: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/grid/0/hadoop/yarn/local/filecache/12/spark-hdp-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.2.0-154/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 17/08/07 03:06:11 INFO ApplicationMaster: Registered signal handlers for [TERM, HUP, INT] 17/08/07 03:06:11 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1502070770869_0012_000001 17/08/07 03:06:12 INFO SecurityManager: Changing view acls to: hrt_qa 17/08/07 03:06:12 INFO SecurityManager: Changing modify acls to: hrt_qa 17/08/07 03:06:12 INFO SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: Set(hrt_qa); users with modify permissions: Set(hrt_qa) 17/08/07 03:06:13 INFO ApplicationMaster: Starting the user application in a separate Thread 17/08/07 03:06:13 INFO ApplicationMaster: Waiting for spark context initialization 17/08/07 03:06:13 INFO ApplicationMaster: Waiting for spark context initialization ... 17/08/07 03:06:14 INFO SparkContext: Running Spark version 1.6.3 17/08/07 03:06:14 INFO SecurityManager: Changing view acls to: hrt_qa 17/08/07 03:06:14 INFO SecurityManager: Changing modify acls to: hrt_qa 17/08/07 03:06:14 INFO SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: Set(hrt_qa); users with modify permissions: Set(hrt_qa) 17/08/07 03:06:14 INFO Utils: Successfully started service 'sparkDriver' on port 53417. 17/08/07 03:06:14 INFO Slf4jLogger: Slf4jLogger started 17/08/07 03:06:14 INFO Remoting: Starting remoting 17/08/07 03:06:14 INFO Remoting: Remoting started; listening on addresses :[akka.ssl.tcp://sparkDriverActorSystem@xxx.xx.xx.xxx:52750] 17/08/07 03:06:15 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 52750. 17/08/07 03:06:15 INFO SparkEnv: Registering MapOutputTracker 17/08/07 03:06:15 INFO SparkEnv: Registering BlockManagerMaster 17/08/07 03:06:15 INFO DiskBlockManager: Created local directory at /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1502070770869_0012/blockmgr-3aa6a838-872b-4644-9f62-e5dda89c6ca7 17/08/07 03:06:15 INFO MemoryStore: MemoryStore started with capacity 480.8 MB 17/08/07 03:06:15 INFO SparkEnv: Registering OutputCommitCoordinator 17/08/07 03:06:15 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 17/08/07 03:06:15 INFO Server: jetty-8.y.z-SNAPSHOT 17/08/07 03:06:15 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:56049 17/08/07 03:06:15 INFO Utils: Successfully started service 'SparkUI' on port 56049. 17/08/07 03:06:15 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://xxx.xx.xx.xxx:56049 17/08/07 03:06:15 INFO YarnClusterScheduler: Created YarnClusterScheduler 17/08/07 03:06:15 INFO SchedulerExtensionServices: Starting Yarn extension services with app application_1502070770869_0012 and attemptId Some(appattempt_1502070770869_0012_000001) 17/08/07 03:06:15 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43180. 17/08/07 03:06:15 INFO NettyBlockTransferService: Server created on 43180 17/08/07 03:06:15 INFO BlockManagerMaster: Trying to register BlockManager 17/08/07 03:06:15 INFO BlockManagerMasterEndpoint: Registering block manager xxx.xx.xx.xxx:43180 with 480.8 MB RAM, BlockManagerId(driver, xxx.xx.xx.xxx, 43180) 17/08/07 03:06:15 INFO BlockManagerMaster: Registered BlockManager 17/08/07 03:06:16 INFO EventLoggingListener: Logging events to hdfs:///spark-history/application_1502070770869_0012_1 17/08/07 03:06:16 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@xxx.xx.xx.xxx:53417) 17/08/07 03:06:16 INFO YarnRMClient: Registering the ApplicationMaster 17/08/07 03:06:16 INFO RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]... 17/08/07 03:06:16 INFO RequestHedgingRMFailoverProxyProvider: Found active RM [rm1] 17/08/07 03:06:16 INFO YarnAllocator: Will request 2 executor containers, each with 1 cores and 1408 MB memory including 384 MB overhead 17/08/07 03:06:16 INFO YarnAllocator: Container request (host: Any, capability: ) 17/08/07 03:06:16 INFO YarnAllocator: Container request (host: Any, capability: ) 17/08/07 03:06:16 INFO ApplicationMaster: Started progress reporter thread with (heartbeat : 5000, initial allocation : 200) intervals 17/08/07 03:06:16 INFO AMRMClientImpl: Received new token for : host6:25454 17/08/07 03:06:16 INFO YarnAllocator: Launching container container_e02_1502070770869_0012_01_000002 for on host host6 17/08/07 03:06:16 INFO YarnAllocator: Launching ExecutorRunnable. driverUrl: spark://CoarseGrainedScheduler@xxx.xx.xx.xxx:53417, executorHostname: host6 17/08/07 03:06:16 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. 17/08/07 03:06:16 INFO ExecutorRunnable: Starting Executor Container 17/08/07 03:06:16 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 17/08/07 03:06:16 INFO ExecutorRunnable: Setting up ContainerLaunchContext 17/08/07 03:06:16 INFO ExecutorRunnable: Preparing Local resources 17/08/07 03:06:16 INFO ExecutorRunnable: Prepared Local resources Map(__spark__.jar -> resource { scheme: "hdfs" host: "mycluster" port: -1 file: "/hdp/apps/2.6.2.0-154/spark/spark-hdp-assembly.jar" } size: 190878097 timestamp: 1502064724925 type: FILE visibility: PUBLIC, sparkr -> resource { scheme: "hdfs" host: "mycluster" port: -1 file: "/user/hrt_qa/.sparkStaging/application_1502070770869_0012/sparkr.zip" } size: 555499 timestamp: 1502075165966 type: ARCHIVE visibility: PRIVATE, __spark_conf__ -> resource { scheme: "hdfs" host: "mycluster" port: -1 file: "/user/hrt_qa/.sparkStaging/application_1502070770869_0012/__spark_conf__5875502619868816278.zip" } size: 144034 timestamp: 1502075166070 type: ARCHIVE visibility: PRIVATE, ml.R -> resource { scheme: "hdfs" host: "mycluster" port: -1 file: "/user/hrt_qa/.sparkStaging/application_1502070770869_0012/ml.R" } size: 1903 timestamp: 1502075165821 type: FILE visibility: PRIVATE) 17/08/07 03:06:17 INFO ExecutorRunnable: =============================================================================== YARN executor launch context: env: CLASSPATH -> {{PWD}}{{PWD}}/__spark_conf__{{PWD}}/__spark__.jar/etc/hadoop/conf/usr/hdp/current/hadoop-client/*/usr/hdp/current/hadoop-client/lib/*/usr/hdp/current/hadoop-hdfs-client/*/usr/hdp/current/hadoop-hdfs-client/lib/*/usr/hdp/current/hadoop-yarn-client/*/usr/hdp/current/hadoop-yarn-client/lib/*/usr/hdp/current/ext/hadoop/*$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/2.6.2.0-154/hadoop/lib/hadoop-lzo-0.6.0.2.6.2.0-154.jar:/etc/hadoop/conf/secure:/usr/hdp/current/ext/hadoop/* SPARK_YARN_CACHE_ARCHIVES -> hdfs://mycluster/user/hrt_qa/.sparkStaging/application_1502070770869_0012/sparkr.zip#sparkr,hdfs://mycluster/user/hrt_qa/.sparkStaging/application_1502070770869_0012/__spark_conf__5875502619868816278.zip#__spark_conf__ SPARK_LOG_URL_STDERR -> https://host6:54321/node/containerlogs/container_e02_1502070770869_0012_01_000002/hrt_qa/stderr?start=-4096 SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1502070770869_0012 SPARK_YARN_CACHE_FILES_FILE_SIZES -> 190878097,1903 SPARK_USER -> hrt_qa SPARK_YARN_CACHE_ARCHIVES_FILE_SIZES -> 555499,144034 SPARK_YARN_CACHE_FILES_VISIBILITIES -> PUBLIC,PRIVATE SPARK_YARN_CACHE_ARCHIVES_TIME_STAMPS -> 1502075165966,1502075166070 SPARK_YARN_MODE -> true SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1502064724925,1502075165821 LD_LIBRARY_PATH -> $LD_LIBRARY_PATH:/usr/lib64/R/lib:/usr/lib/jvm/jre/lib/amd64/server:/usr/lib/jvm/jre/lib/amd64:/usr/lib/jvm/java/lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib:/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64: SPARK_LOG_URL_STDOUT -> https://host6:54321/node/containerlogs/container_e02_1502070770869_0012_01_000002/hrt_qa/stdout?start=-4096 SPARK_YARN_CACHE_ARCHIVES_VISIBILITIES -> PRIVATE,PRIVATE SPARK_YARN_CACHE_FILES -> hdfs://mycluster/hdp/apps/2.6.2.0-154/spark/spark-hdp-assembly.jar#__spark__.jar,hdfs://mycluster/user/hrt_qa/.sparkStaging/application_1502070770869_0012/ml.R#ml.R command: LD_LIBRARY_PATH="/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:$LD_LIBRARY_PATH" {{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms1024m -Xmx1024m -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.authenticate=true' '-Dspark.ssl.trustStore=/etc/security/clientKeys/spark_all.jks' '-Dspark.ssl.keyStore=/etc/security/serverKeys/spark_keystore.jks' '-Dspark.authenticate.enableSaslEncryption=true' '-Dspark.driver.port=53417' '-Dspark.ssl.enabledAlgorithms=TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA' '-Dspark.ssl.enabled=true' '-Dspark.history.ui.port=18080' '-Dspark.ssl.trustStorePassword=clientTrustStorePassword' '-Dspark.ui.port=0' '-Dspark.ssl.protocol=TLS' '-Dspark.ssl.keyPassword=serverKeyPassPassword' '-Dspark.ssl.keyStorePassword=serverStorePassPassword' -Dspark.yarn.app.container.log.dir= org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@xxx.xx.xx.xxx:53417 --executor-id 1 --hostname host6 --cores 1 --app-id application_1502070770869_0012 --user-class-path file:$PWD/__app__.jar 1> /stdout 2> /stderr =============================================================================== 17/08/07 03:06:17 INFO ContainerManagementProtocolProxy: Opening proxy : host6:25454 17/08/07 03:06:17 INFO AMRMClientImpl: Received new token for : host4:25454 17/08/07 03:06:17 INFO YarnAllocator: Launching container container_e02_1502070770869_0012_01_000003 for on host host4 17/08/07 03:06:17 INFO YarnAllocator: Launching ExecutorRunnable. driverUrl: spark://CoarseGrainedScheduler@xxx.xx.xx.xxx:53417, executorHostname: host4 17/08/07 03:06:17 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. 17/08/07 03:06:17 INFO ExecutorRunnable: Starting Executor Container 17/08/07 03:06:17 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 17/08/07 03:06:17 INFO ExecutorRunnable: Setting up ContainerLaunchContext 17/08/07 03:06:17 INFO ExecutorRunnable: Preparing Local resources 17/08/07 03:06:17 INFO ExecutorRunnable: Prepared Local resources Map(__spark__.jar -> resource { scheme: "hdfs" host: "mycluster" port: -1 file: "/hdp/apps/2.6.2.0-154/spark/spark-hdp-assembly.jar" } size: 190878097 timestamp: 1502064724925 type: FILE visibility: PUBLIC, sparkr -> resource { scheme: "hdfs" host: "mycluster" port: -1 file: "/user/hrt_qa/.sparkStaging/application_1502070770869_0012/sparkr.zip" } size: 555499 timestamp: 1502075165966 type: ARCHIVE visibility: PRIVATE, __spark_conf__ -> resource { scheme: "hdfs" host: "mycluster" port: -1 file: "/user/hrt_qa/.sparkStaging/application_1502070770869_0012/__spark_conf__5875502619868816278.zip" } size: 144034 timestamp: 1502075166070 type: ARCHIVE visibility: PRIVATE, ml.R -> resource { scheme: "hdfs" host: "mycluster" port: -1 file: "/user/hrt_qa/.sparkStaging/application_1502070770869_0012/ml.R" } size: 1903 timestamp: 1502075165821 type: FILE visibility: PRIVATE) 17/08/07 03:06:17 INFO ExecutorRunnable: =============================================================================== YARN executor launch context: env: CLASSPATH -> {{PWD}}{{PWD}}/__spark_conf__{{PWD}}/__spark__.jar/etc/hadoop/conf/usr/hdp/current/hadoop-client/*/usr/hdp/current/hadoop-client/lib/*/usr/hdp/current/hadoop-hdfs-client/*/usr/hdp/current/hadoop-hdfs-client/lib/*/usr/hdp/current/hadoop-yarn-client/*/usr/hdp/current/hadoop-yarn-client/lib/*/usr/hdp/current/ext/hadoop/*$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/2.6.2.0-154/hadoop/lib/hadoop-lzo-0.6.0.2.6.2.0-154.jar:/etc/hadoop/conf/secure:/usr/hdp/current/ext/hadoop/* SPARK_YARN_CACHE_ARCHIVES -> hdfs://mycluster/user/hrt_qa/.sparkStaging/application_1502070770869_0012/sparkr.zip#sparkr,hdfs://mycluster/user/hrt_qa/.sparkStaging/application_1502070770869_0012/__spark_conf__5875502619868816278.zip#__spark_conf__ SPARK_LOG_URL_STDERR -> https://host4:54321/node/containerlogs/container_e02_1502070770869_0012_01_000003/hrt_qa/stderr?start=-4096 SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1502070770869_0012 SPARK_YARN_CACHE_FILES_FILE_SIZES -> 190878097,1903 SPARK_USER -> hrt_qa SPARK_YARN_CACHE_ARCHIVES_FILE_SIZES -> 555499,144034 SPARK_YARN_CACHE_FILES_VISIBILITIES -> PUBLIC,PRIVATE SPARK_YARN_CACHE_ARCHIVES_TIME_STAMPS -> 1502075165966,1502075166070 SPARK_YARN_MODE -> true SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1502064724925,1502075165821 LD_LIBRARY_PATH -> $LD_LIBRARY_PATH:/usr/lib64/R/lib:/usr/lib/jvm/jre/lib/amd64/server:/usr/lib/jvm/jre/lib/amd64:/usr/lib/jvm/java/lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib:/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64: SPARK_LOG_URL_STDOUT -> https://host4:54321/node/containerlogs/container_e02_1502070770869_0012_01_000003/hrt_qa/stdout?start=-4096 SPARK_YARN_CACHE_ARCHIVES_VISIBILITIES -> PRIVATE,PRIVATE SPARK_YARN_CACHE_FILES -> hdfs://mycluster/hdp/apps/2.6.2.0-154/spark/spark-hdp-assembly.jar#__spark__.jar,hdfs://mycluster/user/hrt_qa/.sparkStaging/application_1502070770869_0012/ml.R#ml.R command: LD_LIBRARY_PATH="/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:$LD_LIBRARY_PATH" {{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms1024m -Xmx1024m -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.authenticate=true' '-Dspark.ssl.trustStore=/etc/security/clientKeys/spark_all.jks' '-Dspark.ssl.keyStore=/etc/security/serverKeys/spark_keystore.jks' '-Dspark.authenticate.enableSaslEncryption=true' '-Dspark.driver.port=53417' '-Dspark.ssl.enabledAlgorithms=TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA' '-Dspark.ssl.enabled=true' '-Dspark.history.ui.port=18080' '-Dspark.ssl.trustStorePassword=clientTrustStorePassword' '-Dspark.ui.port=0' '-Dspark.ssl.protocol=TLS' '-Dspark.ssl.keyPassword=serverKeyPassPassword' '-Dspark.ssl.keyStorePassword=serverStorePassPassword' -Dspark.yarn.app.container.log.dir= org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@xxx.xx.xx.xxx:53417 --executor-id 2 --hostname host4 --cores 1 --app-id application_1502070770869_0012 --user-class-path file:$PWD/__app__.jar 1> /stdout 2> /stderr =============================================================================== 17/08/07 03:06:17 INFO ContainerManagementProtocolProxy: Opening proxy : host4:25454 17/08/07 03:06:20 INFO YarnClusterSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (host4:52658) with ID 2 17/08/07 03:06:20 INFO BlockManagerMasterEndpoint: Registering block manager host4:33596 with 511.1 MB RAM, BlockManagerId(2, host4, 33596) 17/08/07 03:06:22 INFO AMRMClientImpl: Received new token for : host5:25454 17/08/07 03:06:22 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 0 of them. 17/08/07 03:06:22 INFO YarnClusterSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (host6:49320) with ID 1 17/08/07 03:06:23 INFO BlockManagerMasterEndpoint: Registering block manager host6:46860 with 511.1 MB RAM, BlockManagerId(1, host6, 46860) 17/08/07 03:06:23 INFO YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 17/08/07 03:06:23 INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook done 17/08/07 03:06:23 INFO SparkContext: Starting job: collectPartitions at NativeMethodAccessorImpl.java:-2 17/08/07 03:06:23 INFO DAGScheduler: Got job 0 (collectPartitions at NativeMethodAccessorImpl.java:-2) with 1 output partitions 17/08/07 03:06:23 INFO DAGScheduler: Final stage: ResultStage 0 (collectPartitions at NativeMethodAccessorImpl.java:-2) 17/08/07 03:06:23 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:23 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:23 INFO DAGScheduler: Submitting ResultStage 0 (ParallelCollectionRDD[0] at parallelize at RRDD.scala:460), which has no missing parents 17/08/07 03:06:24 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1344.0 B, free 480.7 MB) 17/08/07 03:06:24 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 890.0 B, free 480.7 MB) 17/08/07 03:06:24 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 890.0 B, free: 480.7 MB) 17/08/07 03:06:24 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:24 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (ParallelCollectionRDD[0] at parallelize at RRDD.scala:460) 17/08/07 03:06:24 INFO YarnClusterScheduler: Adding task set 0.0 with 1 tasks 17/08/07 03:06:24 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, host6, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:24 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on host6:46860 (size: 890.0 B, free: 511.1 MB) 17/08/07 03:06:24 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 562 ms on host6 (1/1) 17/08/07 03:06:24 INFO YarnClusterScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool 17/08/07 03:06:24 INFO DAGScheduler: ResultStage 0 (collectPartitions at NativeMethodAccessorImpl.java:-2) finished in 0.570 s 17/08/07 03:06:24 INFO DAGScheduler: Job 0 finished: collectPartitions at NativeMethodAccessorImpl.java:-2, took 0.962179 s 17/08/07 03:06:26 INFO SparkContext: Starting job: countByValue at StringIndexer.scala:86 17/08/07 03:06:26 INFO DAGScheduler: Registering RDD 8 (countByValue at StringIndexer.scala:86) 17/08/07 03:06:26 INFO DAGScheduler: Got job 1 (countByValue at StringIndexer.scala:86) with 1 output partitions 17/08/07 03:06:26 INFO DAGScheduler: Final stage: ResultStage 2 (countByValue at StringIndexer.scala:86) 17/08/07 03:06:26 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1) 17/08/07 03:06:26 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 1) 17/08/07 03:06:26 INFO DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[8] at countByValue at StringIndexer.scala:86), which has no missing parents 17/08/07 03:06:26 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 14.4 KB, free 480.7 MB) 17/08/07 03:06:26 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 6.0 KB, free 480.7 MB) 17/08/07 03:06:26 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 6.0 KB, free: 480.7 MB) 17/08/07 03:06:26 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:26 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[8] at countByValue at StringIndexer.scala:86) 17/08/07 03:06:26 INFO YarnClusterScheduler: Adding task set 1.0 with 1 tasks 17/08/07 03:06:26 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, host4, partition 0,PROCESS_LOCAL, 16535 bytes) 17/08/07 03:06:26 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on host4:33596 (size: 6.0 KB, free: 511.1 MB) 17/08/07 03:06:28 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 2289 ms on host4 (1/1) 17/08/07 03:06:28 INFO YarnClusterScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool 17/08/07 03:06:28 INFO DAGScheduler: ShuffleMapStage 1 (countByValue at StringIndexer.scala:86) finished in 2.292 s 17/08/07 03:06:28 INFO DAGScheduler: looking for newly runnable stages 17/08/07 03:06:28 INFO DAGScheduler: running: Set() 17/08/07 03:06:28 INFO DAGScheduler: waiting: Set(ResultStage 2) 17/08/07 03:06:28 INFO DAGScheduler: failed: Set() 17/08/07 03:06:28 INFO DAGScheduler: Submitting ResultStage 2 (ShuffledRDD[9] at countByValue at StringIndexer.scala:86), which has no missing parents 17/08/07 03:06:28 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.7 KB, free 480.7 MB) 17/08/07 03:06:28 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1613.0 B, free 480.7 MB) 17/08/07 03:06:28 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 1613.0 B, free: 480.7 MB) 17/08/07 03:06:28 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:28 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (ShuffledRDD[9] at countByValue at StringIndexer.scala:86) 17/08/07 03:06:28 INFO YarnClusterScheduler: Adding task set 2.0 with 1 tasks 17/08/07 03:06:28 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, host4, partition 0,NODE_LOCAL, 1894 bytes) 17/08/07 03:06:28 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on host4:33596 (size: 1613.0 B, free: 511.1 MB) 17/08/07 03:06:28 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to host4:52658 17/08/07 03:06:28 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 173 bytes 17/08/07 03:06:28 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 100 ms on host4 (1/1) 17/08/07 03:06:28 INFO YarnClusterScheduler: Removed TaskSet 2.0, whose tasks have all completed, from pool 17/08/07 03:06:28 INFO DAGScheduler: ResultStage 2 (countByValue at StringIndexer.scala:86) finished in 0.103 s 17/08/07 03:06:28 INFO DAGScheduler: Job 1 finished: countByValue at StringIndexer.scala:86, took 2.482713 s 17/08/07 03:06:29 INFO SparkContext: Starting job: first at LinearRegression.scala:163 17/08/07 03:06:29 INFO DAGScheduler: Registering RDD 12 (map at LinearRegression.scala:161) 17/08/07 03:06:29 INFO DAGScheduler: Got job 2 (first at LinearRegression.scala:163) with 1 output partitions 17/08/07 03:06:29 INFO DAGScheduler: Final stage: ResultStage 4 (first at LinearRegression.scala:163) 17/08/07 03:06:29 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 3) 17/08/07 03:06:29 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 3) 17/08/07 03:06:29 INFO DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[12] at map at LinearRegression.scala:161), which has no missing parents 17/08/07 03:06:29 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 20.7 KB, free 480.7 MB) 17/08/07 03:06:29 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.7 MB) 17/08/07 03:06:29 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:29 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:29 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[12] at map at LinearRegression.scala:161) 17/08/07 03:06:29 INFO YarnClusterScheduler: Adding task set 3.0 with 1 tasks 17/08/07 03:06:29 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, host4, partition 0,PROCESS_LOCAL, 16535 bytes) 17/08/07 03:06:29 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:30 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 614 ms on host4 (1/1) 17/08/07 03:06:30 INFO DAGScheduler: ShuffleMapStage 3 (map at LinearRegression.scala:161) finished in 0.614 s 17/08/07 03:06:30 INFO YarnClusterScheduler: Removed TaskSet 3.0, whose tasks have all completed, from pool 17/08/07 03:06:30 INFO DAGScheduler: looking for newly runnable stages 17/08/07 03:06:30 INFO DAGScheduler: running: Set() 17/08/07 03:06:30 INFO DAGScheduler: waiting: Set(ResultStage 4) 17/08/07 03:06:30 INFO DAGScheduler: failed: Set() 17/08/07 03:06:30 INFO DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[16] at map at LinearRegression.scala:161), which has no missing parents 17/08/07 03:06:30 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 21.2 KB, free 480.7 MB) 17/08/07 03:06:30 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 9.3 KB, free 480.7 MB) 17/08/07 03:06:30 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.3 KB, free: 480.7 MB) 17/08/07 03:06:30 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:30 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[16] at map at LinearRegression.scala:161) 17/08/07 03:06:30 INFO YarnClusterScheduler: Adding task set 4.0 with 1 tasks 17/08/07 03:06:30 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 4, host4, partition 0,NODE_LOCAL, 1894 bytes) 17/08/07 03:06:30 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on host4:33596 (size: 9.3 KB, free: 511.1 MB) 17/08/07 03:06:30 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to host4:52658 17/08/07 03:06:30 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 173 bytes 17/08/07 03:06:30 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 4) in 70 ms on host4 (1/1) 17/08/07 03:06:30 INFO YarnClusterScheduler: Removed TaskSet 4.0, whose tasks have all completed, from pool 17/08/07 03:06:30 INFO DAGScheduler: ResultStage 4 (first at LinearRegression.scala:163) finished in 0.070 s 17/08/07 03:06:30 INFO DAGScheduler: Job 2 finished: first at LinearRegression.scala:163, took 0.715392 s 17/08/07 03:06:30 WARN WeightedLeastSquares: regParam is zero, which might cause numerical instability and overfitting. 17/08/07 03:06:30 INFO SparkContext: Starting job: treeAggregate at WeightedLeastSquares.scala:75 17/08/07 03:06:30 INFO DAGScheduler: Got job 3 (treeAggregate at WeightedLeastSquares.scala:75) with 1 output partitions 17/08/07 03:06:30 INFO DAGScheduler: Final stage: ResultStage 5 (treeAggregate at WeightedLeastSquares.scala:75) 17/08/07 03:06:30 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:30 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:30 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[20] at treeAggregate at WeightedLeastSquares.scala:75), which has no missing parents 17/08/07 03:06:30 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 20.7 KB, free 480.6 MB) 17/08/07 03:06:30 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.6 MB) 17/08/07 03:06:30 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:30 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:30 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[20] at treeAggregate at WeightedLeastSquares.scala:75) 17/08/07 03:06:30 INFO YarnClusterScheduler: Adding task set 5.0 with 1 tasks 17/08/07 03:06:30 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 5, host6, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:30 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on host6:46860 (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:32 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 5) in 1999 ms on host6 (1/1) 17/08/07 03:06:32 INFO DAGScheduler: ResultStage 5 (treeAggregate at WeightedLeastSquares.scala:75) finished in 2.000 s 17/08/07 03:06:32 INFO YarnClusterScheduler: Removed TaskSet 5.0, whose tasks have all completed, from pool 17/08/07 03:06:32 INFO DAGScheduler: Job 3 finished: treeAggregate at WeightedLeastSquares.scala:75, took 2.014754 s 17/08/07 03:06:32 INFO WeightedLeastSquares: Number of instances: 150. 17/08/07 03:06:32 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS 17/08/07 03:06:32 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS 17/08/07 03:06:32 WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK 17/08/07 03:06:32 WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK 17/08/07 03:06:32 INFO SparkContext: Starting job: aggregate at RegressionMetrics.scala:50 17/08/07 03:06:32 INFO DAGScheduler: Got job 4 (aggregate at RegressionMetrics.scala:50) with 1 output partitions 17/08/07 03:06:32 INFO DAGScheduler: Final stage: ResultStage 6 (aggregate at RegressionMetrics.scala:50) 17/08/07 03:06:32 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:32 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:32 INFO DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[24] at map at RegressionMetrics.scala:48), which has no missing parents 17/08/07 03:06:32 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 22.9 KB, free 480.6 MB) 17/08/07 03:06:32 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 10.1 KB, free 480.6 MB) 17/08/07 03:06:32 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 10.1 KB, free: 480.7 MB) 17/08/07 03:06:32 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:32 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 6 (MapPartitionsRDD[24] at map at RegressionMetrics.scala:48) 17/08/07 03:06:32 INFO YarnClusterScheduler: Adding task set 6.0 with 1 tasks 17/08/07 03:06:32 INFO TaskSetManager: Starting task 0.0 in stage 6.0 (TID 6, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:32 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on host4:33596 (size: 10.1 KB, free: 511.1 MB) 17/08/07 03:06:32 INFO TaskSetManager: Finished task 0.0 in stage 6.0 (TID 6) in 194 ms on host4 (1/1) 17/08/07 03:06:32 INFO YarnClusterScheduler: Removed TaskSet 6.0, whose tasks have all completed, from pool 17/08/07 03:06:32 INFO DAGScheduler: ResultStage 6 (aggregate at RegressionMetrics.scala:50) finished in 0.194 s 17/08/07 03:06:32 INFO DAGScheduler: Job 4 finished: aggregate at RegressionMetrics.scala:50, took 0.209884 s 17/08/07 03:06:32 INFO ContextCleaner: Cleaned accumulator 10 17/08/07 03:06:32 INFO SparkContext: Starting job: sum at RegressionMetrics.scala:62 17/08/07 03:06:32 INFO DAGScheduler: Got job 5 (sum at RegressionMetrics.scala:62) with 1 output partitions 17/08/07 03:06:32 INFO DAGScheduler: Final stage: ResultStage 7 (sum at RegressionMetrics.scala:62) 17/08/07 03:06:32 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:32 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:32 INFO DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[25] at map at RegressionMetrics.scala:60), which has no missing parents 17/08/07 03:06:32 INFO BlockManagerInfo: Removed broadcast_4_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.3 KB, free: 480.7 MB) 17/08/07 03:06:32 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 22.4 KB, free 480.6 MB) 17/08/07 03:06:32 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 9.9 KB, free 480.6 MB) 17/08/07 03:06:32 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.9 KB, free: 480.7 MB) 17/08/07 03:06:32 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:32 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 7 (MapPartitionsRDD[25] at map at RegressionMetrics.scala:60) 17/08/07 03:06:32 INFO YarnClusterScheduler: Adding task set 7.0 with 1 tasks 17/08/07 03:06:32 INFO TaskSetManager: Starting task 0.0 in stage 7.0 (TID 7, host6, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:32 INFO BlockManagerInfo: Removed broadcast_4_piece0 on host4:33596 in memory (size: 9.3 KB, free: 511.1 MB) 17/08/07 03:06:32 INFO ContextCleaner: Cleaned accumulator 8 17/08/07 03:06:32 INFO BlockManagerInfo: Removed broadcast_3_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:32 INFO BlockManagerInfo: Removed broadcast_3_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:32 INFO ContextCleaner: Cleaned accumulator 7 17/08/07 03:06:32 INFO ContextCleaner: Cleaned shuffle 1 17/08/07 03:06:32 INFO ContextCleaner: Cleaned accumulator 6 17/08/07 03:06:32 INFO BlockManagerInfo: Removed broadcast_2_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 1613.0 B, free: 480.7 MB) 17/08/07 03:06:32 INFO BlockManagerInfo: Removed broadcast_2_piece0 on host4:33596 in memory (size: 1613.0 B, free: 511.1 MB) 17/08/07 03:06:32 INFO ContextCleaner: Cleaned accumulator 5 17/08/07 03:06:32 INFO BlockManagerInfo: Removed broadcast_1_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 6.0 KB, free: 480.7 MB) 17/08/07 03:06:32 INFO BlockManagerInfo: Removed broadcast_1_piece0 on host4:33596 in memory (size: 6.0 KB, free: 511.1 MB) 17/08/07 03:06:33 INFO ContextCleaner: Cleaned accumulator 4 17/08/07 03:06:33 INFO ContextCleaner: Cleaned shuffle 0 17/08/07 03:06:33 INFO ContextCleaner: Cleaned accumulator 3 17/08/07 03:06:33 INFO BlockManagerInfo: Removed broadcast_0_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 890.0 B, free: 480.7 MB) 17/08/07 03:06:33 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on host6:46860 (size: 9.9 KB, free: 511.1 MB) 17/08/07 03:06:33 INFO BlockManagerInfo: Removed broadcast_0_piece0 on host6:46860 in memory (size: 890.0 B, free: 511.1 MB) 17/08/07 03:06:33 INFO BlockManagerInfo: Removed broadcast_5_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:33 INFO BlockManagerInfo: Removed broadcast_5_piece0 on host6:46860 in memory (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:33 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID 7) in 190 ms on host6 (1/1) 17/08/07 03:06:33 INFO DAGScheduler: ResultStage 7 (sum at RegressionMetrics.scala:62) finished in 0.192 s 17/08/07 03:06:33 INFO YarnClusterScheduler: Removed TaskSet 7.0, whose tasks have all completed, from pool 17/08/07 03:06:33 INFO DAGScheduler: Job 5 finished: sum at RegressionMetrics.scala:62, took 0.204234 s 17/08/07 03:06:33 INFO SparkContext: Starting job: count at LinearRegression.scala:598 17/08/07 03:06:33 INFO DAGScheduler: Registering RDD 28 (count at LinearRegression.scala:598) 17/08/07 03:06:33 INFO DAGScheduler: Got job 6 (count at LinearRegression.scala:598) with 1 output partitions 17/08/07 03:06:33 INFO DAGScheduler: Final stage: ResultStage 9 (count at LinearRegression.scala:598) 17/08/07 03:06:33 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 8) 17/08/07 03:06:33 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 8) 17/08/07 03:06:33 INFO DAGScheduler: Submitting ShuffleMapStage 8 (MapPartitionsRDD[28] at count at LinearRegression.scala:598), which has no missing parents 17/08/07 03:06:33 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 16.9 KB, free 480.7 MB) 17/08/07 03:06:33 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 7.0 KB, free 480.7 MB) 17/08/07 03:06:33 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 7.0 KB, free: 480.7 MB) 17/08/07 03:06:33 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:33 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 8 (MapPartitionsRDD[28] at count at LinearRegression.scala:598) 17/08/07 03:06:33 INFO YarnClusterScheduler: Adding task set 8.0 with 1 tasks 17/08/07 03:06:33 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 8, host4, partition 0,PROCESS_LOCAL, 16535 bytes) 17/08/07 03:06:33 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on host4:33596 (size: 7.0 KB, free: 511.1 MB) 17/08/07 03:06:33 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 8) in 187 ms on host4 (1/1) 17/08/07 03:06:33 INFO DAGScheduler: ShuffleMapStage 8 (count at LinearRegression.scala:598) finished in 0.188 s 17/08/07 03:06:33 INFO DAGScheduler: looking for newly runnable stages 17/08/07 03:06:33 INFO DAGScheduler: running: Set() 17/08/07 03:06:33 INFO DAGScheduler: waiting: Set(ResultStage 9) 17/08/07 03:06:33 INFO DAGScheduler: failed: Set() 17/08/07 03:06:33 INFO YarnClusterScheduler: Removed TaskSet 8.0, whose tasks have all completed, from pool 17/08/07 03:06:33 INFO DAGScheduler: Submitting ResultStage 9 (MapPartitionsRDD[31] at count at LinearRegression.scala:598), which has no missing parents 17/08/07 03:06:33 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 17.8 KB, free 480.6 MB) 17/08/07 03:06:33 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 7.4 KB, free 480.6 MB) 17/08/07 03:06:33 INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 7.4 KB, free: 480.7 MB) 17/08/07 03:06:33 INFO SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:33 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 9 (MapPartitionsRDD[31] at count at LinearRegression.scala:598) 17/08/07 03:06:33 INFO YarnClusterScheduler: Adding task set 9.0 with 1 tasks 17/08/07 03:06:33 INFO TaskSetManager: Starting task 0.0 in stage 9.0 (TID 9, host4, partition 0,NODE_LOCAL, 1999 bytes) 17/08/07 03:06:33 INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on host4:33596 (size: 7.4 KB, free: 511.1 MB) 17/08/07 03:06:33 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to host4:52658 17/08/07 03:06:33 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 2 is 173 bytes 17/08/07 03:06:33 INFO TaskSetManager: Finished task 0.0 in stage 9.0 (TID 9) in 93 ms on host4 (1/1) 17/08/07 03:06:33 INFO YarnClusterScheduler: Removed TaskSet 9.0, whose tasks have all completed, from pool 17/08/07 03:06:33 INFO DAGScheduler: ResultStage 9 (count at LinearRegression.scala:598) finished in 0.094 s 17/08/07 03:06:33 INFO DAGScheduler: Job 6 finished: count at LinearRegression.scala:598, took 0.299874 s 17/08/07 03:06:33 INFO SparkContext: Starting job: first at LinearRegression.scala:616 17/08/07 03:06:33 INFO DAGScheduler: Registering RDD 34 (first at LinearRegression.scala:616) 17/08/07 03:06:33 INFO DAGScheduler: Got job 7 (first at LinearRegression.scala:616) with 1 output partitions 17/08/07 03:06:33 INFO DAGScheduler: Final stage: ResultStage 11 (first at LinearRegression.scala:616) 17/08/07 03:06:33 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 10) 17/08/07 03:06:33 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 10) 17/08/07 03:06:33 INFO DAGScheduler: Submitting ShuffleMapStage 10 (MapPartitionsRDD[34] at first at LinearRegression.scala:616), which has no missing parents 17/08/07 03:06:33 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 27.2 KB, free 480.6 MB) 17/08/07 03:06:33 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 12.0 KB, free 480.6 MB) 17/08/07 03:06:33 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 12.0 KB, free: 480.7 MB) 17/08/07 03:06:33 INFO SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:33 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 10 (MapPartitionsRDD[34] at first at LinearRegression.scala:616) 17/08/07 03:06:33 INFO YarnClusterScheduler: Adding task set 10.0 with 1 tasks 17/08/07 03:06:33 INFO TaskSetManager: Starting task 0.0 in stage 10.0 (TID 10, host4, partition 0,PROCESS_LOCAL, 16535 bytes) 17/08/07 03:06:33 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on host4:33596 (size: 12.0 KB, free: 511.1 MB) 17/08/07 03:06:34 INFO TaskSetManager: Finished task 0.0 in stage 10.0 (TID 10) in 171 ms on host4 (1/1) 17/08/07 03:06:34 INFO DAGScheduler: ShuffleMapStage 10 (first at LinearRegression.scala:616) finished in 0.172 s 17/08/07 03:06:34 INFO YarnClusterScheduler: Removed TaskSet 10.0, whose tasks have all completed, from pool 17/08/07 03:06:34 INFO DAGScheduler: looking for newly runnable stages 17/08/07 03:06:34 INFO DAGScheduler: running: Set() 17/08/07 03:06:34 INFO DAGScheduler: waiting: Set(ResultStage 11) 17/08/07 03:06:34 INFO DAGScheduler: failed: Set() 17/08/07 03:06:34 INFO DAGScheduler: Submitting ResultStage 11 (MapPartitionsRDD[38] at first at LinearRegression.scala:616), which has no missing parents 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 28.2 KB, free 480.6 MB) 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 12.4 KB, free 480.6 MB) 17/08/07 03:06:34 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 12.4 KB, free: 480.7 MB) 17/08/07 03:06:34 INFO SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:34 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 11 (MapPartitionsRDD[38] at first at LinearRegression.scala:616) 17/08/07 03:06:34 INFO YarnClusterScheduler: Adding task set 11.0 with 1 tasks 17/08/07 03:06:34 INFO TaskSetManager: Starting task 0.0 in stage 11.0 (TID 11, host4, partition 0,NODE_LOCAL, 1999 bytes) 17/08/07 03:06:34 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on host4:33596 (size: 12.4 KB, free: 511.1 MB) 17/08/07 03:06:34 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 3 to host4:52658 17/08/07 03:06:34 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 3 is 173 bytes 17/08/07 03:06:34 INFO TaskSetManager: Finished task 0.0 in stage 11.0 (TID 11) in 63 ms on host4 (1/1) 17/08/07 03:06:34 INFO YarnClusterScheduler: Removed TaskSet 11.0, whose tasks have all completed, from pool 17/08/07 03:06:34 INFO DAGScheduler: ResultStage 11 (first at LinearRegression.scala:616) finished in 0.064 s 17/08/07 03:06:34 INFO DAGScheduler: Job 7 finished: first at LinearRegression.scala:616, took 0.256766 s 17/08/07 03:06:34 INFO SparkContext: Starting job: dfToCols at NativeMethodAccessorImpl.java:-2 17/08/07 03:06:34 INFO DAGScheduler: Got job 8 (dfToCols at NativeMethodAccessorImpl.java:-2) with 1 output partitions 17/08/07 03:06:34 INFO DAGScheduler: Final stage: ResultStage 12 (dfToCols at NativeMethodAccessorImpl.java:-2) 17/08/07 03:06:34 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:34 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:34 INFO DAGScheduler: Submitting ResultStage 12 (MapPartitionsRDD[41] at dfToCols at NativeMethodAccessorImpl.java:-2), which has no missing parents 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 22.8 KB, free 480.5 MB) 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 10.2 KB, free 480.5 MB) 17/08/07 03:06:34 INFO BlockManagerInfo: Added broadcast_12_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 10.2 KB, free: 480.7 MB) 17/08/07 03:06:34 INFO SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:34 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 12 (MapPartitionsRDD[41] at dfToCols at NativeMethodAccessorImpl.java:-2) 17/08/07 03:06:34 INFO YarnClusterScheduler: Adding task set 12.0 with 1 tasks 17/08/07 03:06:34 INFO TaskSetManager: Starting task 0.0 in stage 12.0 (TID 12, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:34 INFO BlockManagerInfo: Added broadcast_12_piece0 in memory on host4:33596 (size: 10.2 KB, free: 511.1 MB) 17/08/07 03:06:34 INFO TaskSetManager: Finished task 0.0 in stage 12.0 (TID 12) in 81 ms on host4 (1/1) 17/08/07 03:06:34 INFO YarnClusterScheduler: Removed TaskSet 12.0, whose tasks have all completed, from pool 17/08/07 03:06:34 INFO DAGScheduler: ResultStage 12 (dfToCols at NativeMethodAccessorImpl.java:-2) finished in 0.082 s 17/08/07 03:06:34 INFO DAGScheduler: Job 8 finished: dfToCols at NativeMethodAccessorImpl.java:-2, took 0.091532 s 17/08/07 03:06:34 INFO SparkContext: Starting job: countByValue at StringIndexer.scala:86 17/08/07 03:06:34 INFO DAGScheduler: Registering RDD 47 (countByValue at StringIndexer.scala:86) 17/08/07 03:06:34 INFO DAGScheduler: Got job 9 (countByValue at StringIndexer.scala:86) with 1 output partitions 17/08/07 03:06:34 INFO DAGScheduler: Final stage: ResultStage 14 (countByValue at StringIndexer.scala:86) 17/08/07 03:06:34 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 13) 17/08/07 03:06:34 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 13) 17/08/07 03:06:34 INFO DAGScheduler: Submitting ShuffleMapStage 13 (MapPartitionsRDD[47] at countByValue at StringIndexer.scala:86), which has no missing parents 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 15.5 KB, free 480.5 MB) 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 6.4 KB, free 480.5 MB) 17/08/07 03:06:34 INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 6.4 KB, free: 480.7 MB) 17/08/07 03:06:34 INFO SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:34 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 13 (MapPartitionsRDD[47] at countByValue at StringIndexer.scala:86) 17/08/07 03:06:34 INFO YarnClusterScheduler: Adding task set 13.0 with 1 tasks 17/08/07 03:06:34 INFO TaskSetManager: Starting task 0.0 in stage 13.0 (TID 13, host6, partition 0,PROCESS_LOCAL, 16535 bytes) 17/08/07 03:06:34 INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on host6:46860 (size: 6.4 KB, free: 511.1 MB) 17/08/07 03:06:34 INFO TaskSetManager: Finished task 0.0 in stage 13.0 (TID 13) in 185 ms on host6 (1/1) 17/08/07 03:06:34 INFO DAGScheduler: ShuffleMapStage 13 (countByValue at StringIndexer.scala:86) finished in 0.185 s 17/08/07 03:06:34 INFO YarnClusterScheduler: Removed TaskSet 13.0, whose tasks have all completed, from pool 17/08/07 03:06:34 INFO DAGScheduler: looking for newly runnable stages 17/08/07 03:06:34 INFO DAGScheduler: running: Set() 17/08/07 03:06:34 INFO DAGScheduler: waiting: Set(ResultStage 14) 17/08/07 03:06:34 INFO DAGScheduler: failed: Set() 17/08/07 03:06:34 INFO DAGScheduler: Submitting ResultStage 14 (ShuffledRDD[48] at countByValue at StringIndexer.scala:86), which has no missing parents 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_14 stored as values in memory (estimated size 2.7 KB, free 480.5 MB) 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 1607.0 B, free 480.5 MB) 17/08/07 03:06:34 INFO BlockManagerInfo: Added broadcast_14_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 1607.0 B, free: 480.7 MB) 17/08/07 03:06:34 INFO SparkContext: Created broadcast 14 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:34 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 14 (ShuffledRDD[48] at countByValue at StringIndexer.scala:86) 17/08/07 03:06:34 INFO YarnClusterScheduler: Adding task set 14.0 with 1 tasks 17/08/07 03:06:34 INFO TaskSetManager: Starting task 0.0 in stage 14.0 (TID 14, host6, partition 0,NODE_LOCAL, 1894 bytes) 17/08/07 03:06:34 INFO BlockManagerInfo: Added broadcast_14_piece0 in memory on host6:46860 (size: 1607.0 B, free: 511.1 MB) 17/08/07 03:06:34 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 4 to host6:49320 17/08/07 03:06:34 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 4 is 173 bytes 17/08/07 03:06:34 INFO TaskSetManager: Finished task 0.0 in stage 14.0 (TID 14) in 73 ms on host6 (1/1) 17/08/07 03:06:34 INFO YarnClusterScheduler: Removed TaskSet 14.0, whose tasks have all completed, from pool 17/08/07 03:06:34 INFO DAGScheduler: ResultStage 14 (countByValue at StringIndexer.scala:86) finished in 0.075 s 17/08/07 03:06:34 INFO DAGScheduler: Job 9 finished: countByValue at StringIndexer.scala:86, took 0.286932 s 17/08/07 03:06:34 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:271 17/08/07 03:06:34 INFO DAGScheduler: Got job 10 (treeAggregate at LogisticRegression.scala:271) with 1 output partitions 17/08/07 03:06:34 INFO DAGScheduler: Final stage: ResultStage 15 (treeAggregate at LogisticRegression.scala:271) 17/08/07 03:06:34 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:34 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:34 INFO DAGScheduler: Submitting ResultStage 15 (MapPartitionsRDD[56] at treeAggregate at LogisticRegression.scala:271), which has no missing parents 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_15 stored as values in memory (estimated size 21.0 KB, free 480.5 MB) 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.5 MB) 17/08/07 03:06:34 INFO BlockManagerInfo: Added broadcast_15_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:34 INFO SparkContext: Created broadcast 15 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:34 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 15 (MapPartitionsRDD[56] at treeAggregate at LogisticRegression.scala:271) 17/08/07 03:06:34 INFO YarnClusterScheduler: Adding task set 15.0 with 1 tasks 17/08/07 03:06:34 INFO TaskSetManager: Starting task 0.0 in stage 15.0 (TID 15, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:34 INFO BlockManagerInfo: Added broadcast_15_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:34 INFO BlockManagerInfo: Added rdd_52_0 in memory on host4:33596 (size: 8.2 KB, free: 511.1 MB) 17/08/07 03:06:35 INFO TaskSetManager: Finished task 0.0 in stage 15.0 (TID 15) in 160 ms on host4 (1/1) 17/08/07 03:06:35 INFO DAGScheduler: ResultStage 15 (treeAggregate at LogisticRegression.scala:271) finished in 0.160 s 17/08/07 03:06:35 INFO YarnClusterScheduler: Removed TaskSet 15.0, whose tasks have all completed, from pool 17/08/07 03:06:35 INFO DAGScheduler: Job 10 finished: treeAggregate at LogisticRegression.scala:271, took 0.177152 s 17/08/07 03:06:35 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:35 INFO DAGScheduler: Got job 11 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:35 INFO DAGScheduler: Final stage: ResultStage 16 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:35 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:35 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:35 INFO DAGScheduler: Submitting ResultStage 16 (MapPartitionsRDD[57] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:35 INFO MemoryStore: Block broadcast_16 stored as values in memory (estimated size 21.0 KB, free 480.5 MB) 17/08/07 03:06:35 INFO MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.4 MB) 17/08/07 03:06:35 INFO BlockManagerInfo: Added broadcast_16_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:35 INFO SparkContext: Created broadcast 16 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:35 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 16 (MapPartitionsRDD[57] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:35 INFO YarnClusterScheduler: Adding task set 16.0 with 1 tasks 17/08/07 03:06:35 INFO TaskSetManager: Starting task 0.0 in stage 16.0 (TID 16, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:35 INFO BlockManagerInfo: Added broadcast_16_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:35 INFO TaskSetManager: Finished task 0.0 in stage 16.0 (TID 16) in 50 ms on host4 (1/1) 17/08/07 03:06:35 INFO DAGScheduler: ResultStage 16 (treeAggregate at LogisticRegression.scala:1018) finished in 0.052 s 17/08/07 03:06:35 INFO YarnClusterScheduler: Removed TaskSet 16.0, whose tasks have all completed, from pool 17/08/07 03:06:35 INFO DAGScheduler: Job 11 finished: treeAggregate at LogisticRegression.scala:1018, took 0.062686 s 17/08/07 03:06:35 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:35 INFO DAGScheduler: Got job 12 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:35 INFO DAGScheduler: Final stage: ResultStage 17 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:35 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:35 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:35 INFO DAGScheduler: Submitting ResultStage 17 (MapPartitionsRDD[58] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:35 INFO MemoryStore: Block broadcast_17 stored as values in memory (estimated size 21.0 KB, free 480.4 MB) 17/08/07 03:06:35 INFO MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.4 MB) 17/08/07 03:06:35 INFO BlockManagerInfo: Added broadcast_17_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:35 INFO SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:35 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 17 (MapPartitionsRDD[58] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:35 INFO YarnClusterScheduler: Adding task set 17.0 with 1 tasks 17/08/07 03:06:35 INFO TaskSetManager: Starting task 0.0 in stage 17.0 (TID 17, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:35 INFO BlockManagerInfo: Added broadcast_17_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:35 INFO TaskSetManager: Finished task 0.0 in stage 17.0 (TID 17) in 35 ms on host4 (1/1) 17/08/07 03:06:35 INFO YarnClusterScheduler: Removed TaskSet 17.0, whose tasks have all completed, from pool 17/08/07 03:06:35 INFO DAGScheduler: ResultStage 17 (treeAggregate at LogisticRegression.scala:1018) finished in 0.037 s 17/08/07 03:06:35 INFO DAGScheduler: Job 12 finished: treeAggregate at LogisticRegression.scala:1018, took 0.043505 s 17/08/07 03:06:35 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:35 INFO DAGScheduler: Got job 13 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:35 INFO DAGScheduler: Final stage: ResultStage 18 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:35 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:35 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:35 INFO DAGScheduler: Submitting ResultStage 18 (MapPartitionsRDD[59] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:35 INFO MemoryStore: Block broadcast_18 stored as values in memory (estimated size 21.0 KB, free 480.4 MB) 17/08/07 03:06:35 INFO MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.4 MB) 17/08/07 03:06:35 INFO BlockManagerInfo: Added broadcast_18_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:35 INFO SparkContext: Created broadcast 18 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:35 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 18 (MapPartitionsRDD[59] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:35 INFO YarnClusterScheduler: Adding task set 18.0 with 1 tasks 17/08/07 03:06:35 INFO TaskSetManager: Starting task 0.0 in stage 18.0 (TID 18, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:35 INFO BlockManagerInfo: Added broadcast_18_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:35 INFO TaskSetManager: Finished task 0.0 in stage 18.0 (TID 18) in 36 ms on host4 (1/1) 17/08/07 03:06:35 INFO YarnClusterScheduler: Removed TaskSet 18.0, whose tasks have all completed, from pool 17/08/07 03:06:35 INFO DAGScheduler: ResultStage 18 (treeAggregate at LogisticRegression.scala:1018) finished in 0.038 s 17/08/07 03:06:35 INFO DAGScheduler: Job 13 finished: treeAggregate at LogisticRegression.scala:1018, took 0.045476 s 17/08/07 03:06:35 INFO StrongWolfeLineSearch: Line search t: 0.3451057091737686 fval: 0.8515289038516817 rhs: 0.693144282897075 cdd: 0.9370915088741227 17/08/07 03:06:35 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:35 INFO DAGScheduler: Got job 14 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:35 INFO DAGScheduler: Final stage: ResultStage 19 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:35 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:35 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:35 INFO DAGScheduler: Submitting ResultStage 19 (MapPartitionsRDD[60] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:35 INFO MemoryStore: Block broadcast_19 stored as values in memory (estimated size 21.0 KB, free 480.4 MB) 17/08/07 03:06:35 INFO MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.4 MB) 17/08/07 03:06:35 INFO BlockManagerInfo: Added broadcast_19_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:35 INFO SparkContext: Created broadcast 19 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:35 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 19 (MapPartitionsRDD[60] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:35 INFO YarnClusterScheduler: Adding task set 19.0 with 1 tasks 17/08/07 03:06:35 INFO TaskSetManager: Starting task 0.0 in stage 19.0 (TID 19, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_19_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 19.0 (TID 19) in 29 ms on host4 (1/1) 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 19.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 19 (treeAggregate at LogisticRegression.scala:1018) finished in 0.030 s 17/08/07 03:06:36 INFO DAGScheduler: Job 14 finished: treeAggregate at LogisticRegression.scala:1018, took 0.037922 s 17/08/07 03:06:36 INFO StrongWolfeLineSearch: Line search t: 0.03451057091737686 fval: 0.6922457704980421 rhs: 0.6931468907936588 cdd: 0.031645061352024675 17/08/07 03:06:36 INFO LBFGS: Step Size: 0.03451 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.692246 (rel: 0.00130) 0.135460 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 15 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 20 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 20 (MapPartitionsRDD[61] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_20 stored as values in memory (estimated size 21.0 KB, free 480.3 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.3 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_20_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 20 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 20 (MapPartitionsRDD[61] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 20.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 20.0 (TID 20, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_20_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 20.0 (TID 20) in 34 ms on host4 (1/1) 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 20.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 20 (treeAggregate at LogisticRegression.scala:1018) finished in 0.035 s 17/08/07 03:06:36 INFO DAGScheduler: Job 15 finished: treeAggregate at LogisticRegression.scala:1018, took 0.043150 s 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.692014 (rel: 0.000334) 0.0581071 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 16 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 21 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 21 (MapPartitionsRDD[62] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_21 stored as values in memory (estimated size 21.0 KB, free 480.3 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.3 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_21_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 21 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 21 (MapPartitionsRDD[62] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 21.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 21.0 (TID 21, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_21_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 21.0 (TID 21) in 35 ms on host4 (1/1) 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 21.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 21 (treeAggregate at LogisticRegression.scala:1018) finished in 0.037 s 17/08/07 03:06:36 INFO DAGScheduler: Job 16 finished: treeAggregate at LogisticRegression.scala:1018, took 0.046777 s 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.691900 (rel: 0.000165) 0.0597816 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 17 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 22 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 22 (MapPartitionsRDD[63] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_22 stored as values in memory (estimated size 21.0 KB, free 480.3 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_22_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.3 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_22_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 22 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 22 (MapPartitionsRDD[63] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 22.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 22.0 (TID 22, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_22_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 22.0 (TID 22) in 36 ms on host4 (1/1) 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 22.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 22 (treeAggregate at LogisticRegression.scala:1018) finished in 0.037 s 17/08/07 03:06:36 INFO DAGScheduler: Job 17 finished: treeAggregate at LogisticRegression.scala:1018, took 0.051474 s 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.691110 (rel: 0.00114) 0.163658 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 18 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 23 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 23 (MapPartitionsRDD[64] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_23 stored as values in memory (estimated size 21.0 KB, free 480.2 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_23_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.2 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_23_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 23 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 23 (MapPartitionsRDD[64] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 23.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 23.0 (TID 23, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_23_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 23.0 (TID 23) in 31 ms on host4 (1/1) 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 23.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 23 (treeAggregate at LogisticRegression.scala:1018) finished in 0.032 s 17/08/07 03:06:36 INFO DAGScheduler: Job 18 finished: treeAggregate at LogisticRegression.scala:1018, took 0.044479 s 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.689656 (rel: 0.00210) 0.304356 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 19 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 24 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 24 (MapPartitionsRDD[65] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_24 stored as values in memory (estimated size 21.0 KB, free 480.2 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_24_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.2 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_24_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 24 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 24 (MapPartitionsRDD[65] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 24.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 24.0 (TID 24, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_24_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 24.0 (TID 24) in 35 ms on host4 (1/1) 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 24.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 24 (treeAggregate at LogisticRegression.scala:1018) finished in 0.037 s 17/08/07 03:06:36 INFO DAGScheduler: Job 19 finished: treeAggregate at LogisticRegression.scala:1018, took 0.048621 s 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.686200 (rel: 0.00501) 0.488087 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 20 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 25 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 25 (MapPartitionsRDD[66] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_25 stored as values in memory (estimated size 21.0 KB, free 480.2 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_25_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.2 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_25_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 25 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 25 (MapPartitionsRDD[66] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 25.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 25.0 (TID 25, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_25_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 25.0 (TID 25) in 39 ms on host4 (1/1) 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 25.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 25 (treeAggregate at LogisticRegression.scala:1018) finished in 0.039 s 17/08/07 03:06:36 INFO DAGScheduler: Job 20 finished: treeAggregate at LogisticRegression.scala:1018, took 0.051258 s 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.680981 (rel: 0.00761) 0.558109 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 21 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 26 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 26 (MapPartitionsRDD[67] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_26 stored as values in memory (estimated size 21.0 KB, free 480.2 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_26_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.2 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_26_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 26 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 26 (MapPartitionsRDD[67] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 26.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 26.0 (TID 26, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_26_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 26.0 (TID 26) in 34 ms on host4 (1/1) 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 26.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 26 (treeAggregate at LogisticRegression.scala:1018) finished in 0.034 s 17/08/07 03:06:36 INFO DAGScheduler: Job 21 finished: treeAggregate at LogisticRegression.scala:1018, took 0.041441 s 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.675851 (rel: 0.00753) 0.368811 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 22 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 27 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 27 (MapPartitionsRDD[68] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_27 stored as values in memory (estimated size 21.0 KB, free 480.1 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_27_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.1 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_27_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 27 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 27 (MapPartitionsRDD[68] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 27.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 27.0 (TID 27, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_27_piece0 in memory on host4:33596 (size: 9.0 KB, free: 510.9 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 27.0 (TID 27) in 34 ms on host4 (1/1) 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 27.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 27 (treeAggregate at LogisticRegression.scala:1018) finished in 0.035 s 17/08/07 03:06:36 INFO DAGScheduler: Job 22 finished: treeAggregate at LogisticRegression.scala:1018, took 0.042933 s 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.673948 (rel: 0.00282) 0.101578 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 23 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 28 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 28 (MapPartitionsRDD[69] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_28 stored as values in memory (estimated size 21.0 KB, free 480.1 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_28_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.1 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 9 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_28_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 28 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 28 (MapPartitionsRDD[69] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 28.0 with 1 tasks 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_27_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 28.0 (TID 28, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_27_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 61 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_26_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_26_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 60 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_25_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_25_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_28_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 59 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_24_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_24_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 58 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 28.0 (TID 28) in 30 ms on host4 (1/1) 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 28.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 28 (treeAggregate at LogisticRegression.scala:1018) finished in 0.031 s 17/08/07 03:06:36 INFO DAGScheduler: Job 23 finished: treeAggregate at LogisticRegression.scala:1018, took 0.085001 s 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_23_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.673716 (rel: 0.000344) 0.0280276 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_23_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 57 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_22_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 24 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 29 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 29 (MapPartitionsRDD[70] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_22_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_29 stored as values in memory (estimated size 21.0 KB, free 480.2 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 56 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_21_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_29_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.3 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_21_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_29_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 29 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 29 (MapPartitionsRDD[70] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 29.0 with 1 tasks 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 55 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 29.0 (TID 29, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_20_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_20_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 54 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_19_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_19_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 53 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_18_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_29_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_18_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 52 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_17_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_17_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 51 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_16_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 29.0 (TID 29) in 55 ms on host4 (1/1) 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 29.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 29 (treeAggregate at LogisticRegression.scala:1018) finished in 0.056 s 17/08/07 03:06:36 INFO DAGScheduler: Job 24 finished: treeAggregate at LogisticRegression.scala:1018, took 0.065416 s 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.673676 (rel: 5.92e-05) 0.0446388 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_16_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 50 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_15_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 25 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 30 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_15_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 30 (MapPartitionsRDD[71] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_30 stored as values in memory (estimated size 21.0 KB, free 480.4 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 49 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_14_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 1607.0 B, free: 480.7 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_30_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.4 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_30_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 30 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 30 (MapPartitionsRDD[71] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 30.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 30.0 (TID 30, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_14_piece0 on host6:46860 in memory (size: 1607.0 B, free: 511.1 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 42 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_13_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 6.4 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_13_piece0 on host6:46860 in memory (size: 6.4 KB, free: 511.1 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 41 17/08/07 03:06:36 INFO ContextCleaner: Cleaned shuffle 4 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 40 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 39 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 38 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_12_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 10.2 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_12_piece0 on host4:33596 in memory (size: 10.2 KB, free: 511.1 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 37 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 36 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_11_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 12.4 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_30_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_11_piece0 on host4:33596 in memory (size: 12.4 KB, free: 511.1 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 35 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_10_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 12.0 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_10_piece0 on host4:33596 in memory (size: 12.0 KB, free: 511.1 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 34 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 30.0 (TID 30) in 31 ms on host4 (1/1) 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 30.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO ContextCleaner: Cleaned shuffle 3 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 33 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 32 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 31 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 30 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 29 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 28 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 27 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 26 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 25 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 30 (treeAggregate at LogisticRegression.scala:1018) finished in 0.034 s 17/08/07 03:06:36 INFO DAGScheduler: Job 25 finished: treeAggregate at LogisticRegression.scala:1018, took 0.040013 s 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_9_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 7.4 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.673602 (rel: 0.000110) 0.0639475 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_9_piece0 on host4:33596 in memory (size: 7.4 KB, free: 511.1 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 24 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_8_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 7.0 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_8_piece0 on host4:33596 in memory (size: 7.0 KB, free: 511.1 MB) 17/08/07 03:06:36 INFO DAGScheduler: Got job 26 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 31 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 31 (MapPartitionsRDD[72] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 23 17/08/07 03:06:36 INFO ContextCleaner: Cleaned shuffle 2 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 22 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 21 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 20 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 19 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 18 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 17 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 16 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 15 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 14 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_31 stored as values in memory (estimated size 21.0 KB, free 480.6 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_7_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.9 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_31_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.6 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_31_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 31 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 31 (MapPartitionsRDD[72] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 31.0 with 1 tasks 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_7_piece0 on host6:46860 in memory (size: 9.9 KB, free: 511.1 MB) 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 31.0 (TID 31, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 13 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_6_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 10.1 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Removed broadcast_6_piece0 on host4:33596 in memory (size: 10.1 KB, free: 511.1 MB) 17/08/07 03:06:36 INFO ContextCleaner: Cleaned accumulator 12 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_31_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 31.0 (TID 31) in 28 ms on host4 (1/1) 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 31 (treeAggregate at LogisticRegression.scala:1018) finished in 0.028 s 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 31.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: Job 26 finished: treeAggregate at LogisticRegression.scala:1018, took 0.034806 s 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.673360 (rel: 0.000359) 0.102805 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 27 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 32 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 32 (MapPartitionsRDD[73] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_32 stored as values in memory (estimated size 21.0 KB, free 480.6 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_32_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.6 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_32_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 32 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 32 (MapPartitionsRDD[73] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 32.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 32.0 (TID 32, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_32_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 32.0 (TID 32) in 28 ms on host4 (1/1) 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 32 (treeAggregate at LogisticRegression.scala:1018) finished in 0.029 s 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 32.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: Job 27 finished: treeAggregate at LogisticRegression.scala:1018, took 0.036764 s 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.672721 (rel: 0.000949) 0.168360 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 28 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 33 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 33 (MapPartitionsRDD[74] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_33 stored as values in memory (estimated size 21.0 KB, free 480.6 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_33_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.6 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_33_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 33 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 33 (MapPartitionsRDD[74] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 33.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 33.0 (TID 33, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_33_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 33.0 (TID 33) in 27 ms on host4 (1/1) 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 33 (treeAggregate at LogisticRegression.scala:1018) finished in 0.028 s 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 33.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: Job 28 finished: treeAggregate at LogisticRegression.scala:1018, took 0.035385 s 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.671019 (rel: 0.00253) 0.281088 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 29 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 34 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 34 (MapPartitionsRDD[75] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_34 stored as values in memory (estimated size 21.0 KB, free 480.6 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_34_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.5 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_34_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 34 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 34 (MapPartitionsRDD[75] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 34.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 34.0 (TID 34, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_34_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 34.0 (TID 34) in 27 ms on host4 (1/1) 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 34.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 34 (treeAggregate at LogisticRegression.scala:1018) finished in 0.028 s 17/08/07 03:06:36 INFO DAGScheduler: Job 29 finished: treeAggregate at LogisticRegression.scala:1018, took 0.033981 s 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.666658 (rel: 0.00650) 0.464232 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 30 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 35 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 35 (MapPartitionsRDD[76] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_35 stored as values in memory (estimated size 21.0 KB, free 480.5 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_35_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.5 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_35_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 35 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 35 (MapPartitionsRDD[76] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 35.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 35.0 (TID 35, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_35_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 35.0 (TID 35) in 24 ms on host4 (1/1) 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 35.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 35 (treeAggregate at LogisticRegression.scala:1018) finished in 0.025 s 17/08/07 03:06:36 INFO DAGScheduler: Job 30 finished: treeAggregate at LogisticRegression.scala:1018, took 0.029946 s 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.656099 (rel: 0.0158) 0.722391 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 31 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 36 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 36 (MapPartitionsRDD[77] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_36 stored as values in memory (estimated size 21.0 KB, free 480.5 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_36_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.5 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_36_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 36 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 36 (MapPartitionsRDD[77] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 36.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 36.0 (TID 36, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_36_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 36.0 (TID 36) in 23 ms on host4 (1/1) 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 36 (treeAggregate at LogisticRegression.scala:1018) finished in 0.025 s 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 36.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: Job 31 finished: treeAggregate at LogisticRegression.scala:1018, took 0.029531 s 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.633704 (rel: 0.0341) 0.968845 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 32 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 37 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 37 (MapPartitionsRDD[78] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_37 stored as values in memory (estimated size 21.0 KB, free 480.5 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_37_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.5 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_37_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 37 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 37 (MapPartitionsRDD[78] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 37.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 37.0 (TID 37, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_37_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 37.0 (TID 37) in 23 ms on host4 (1/1) 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 37.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 37 (treeAggregate at LogisticRegression.scala:1018) finished in 0.024 s 17/08/07 03:06:36 INFO DAGScheduler: Job 32 finished: treeAggregate at LogisticRegression.scala:1018, took 0.029869 s 17/08/07 03:06:36 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:36 INFO LBFGS: Val and Grad Norm: 0.598308 (rel: 0.0559) 0.993368 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 33 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 38 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 38 (MapPartitionsRDD[79] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_38 stored as values in memory (estimated size 21.0 KB, free 480.4 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_38_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.4 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_38_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 38 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 38 (MapPartitionsRDD[79] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 38.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 38.0 (TID 38, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_38_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:36 INFO TaskSetManager: Finished task 0.0 in stage 38.0 (TID 38) in 23 ms on host4 (1/1) 17/08/07 03:06:36 INFO YarnClusterScheduler: Removed TaskSet 38.0, whose tasks have all completed, from pool 17/08/07 03:06:36 INFO DAGScheduler: ResultStage 38 (treeAggregate at LogisticRegression.scala:1018) finished in 0.024 s 17/08/07 03:06:36 INFO DAGScheduler: Job 33 finished: treeAggregate at LogisticRegression.scala:1018, took 0.030786 s 17/08/07 03:06:36 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:36 INFO DAGScheduler: Got job 34 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:36 INFO DAGScheduler: Final stage: ResultStage 39 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:36 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:36 INFO DAGScheduler: Submitting ResultStage 39 (MapPartitionsRDD[80] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_39 stored as values in memory (estimated size 21.0 KB, free 480.4 MB) 17/08/07 03:06:36 INFO MemoryStore: Block broadcast_39_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.4 MB) 17/08/07 03:06:36 INFO BlockManagerInfo: Added broadcast_39_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:36 INFO SparkContext: Created broadcast 39 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 39 (MapPartitionsRDD[80] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:36 INFO YarnClusterScheduler: Adding task set 39.0 with 1 tasks 17/08/07 03:06:36 INFO TaskSetManager: Starting task 0.0 in stage 39.0 (TID 39, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_39_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO TaskSetManager: Finished task 0.0 in stage 39.0 (TID 39) in 21 ms on host4 (1/1) 17/08/07 03:06:37 INFO DAGScheduler: ResultStage 39 (treeAggregate at LogisticRegression.scala:1018) finished in 0.023 s 17/08/07 03:06:37 INFO YarnClusterScheduler: Removed TaskSet 39.0, whose tasks have all completed, from pool 17/08/07 03:06:37 INFO DAGScheduler: Job 34 finished: treeAggregate at LogisticRegression.scala:1018, took 0.028575 s 17/08/07 03:06:37 INFO StrongWolfeLineSearch: Line search t: 0.32104970417468437 fval: 0.5885916676127432 rhs: 0.5983063903934789 cdd: -3.188052198979663E-4 17/08/07 03:06:37 INFO LBFGS: Step Size: 0.3210 17/08/07 03:06:37 INFO LBFGS: Val and Grad Norm: 0.588592 (rel: 0.0162) 0.290944 17/08/07 03:06:37 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:37 INFO DAGScheduler: Got job 35 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:37 INFO DAGScheduler: Final stage: ResultStage 40 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:37 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:37 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:37 INFO DAGScheduler: Submitting ResultStage 40 (MapPartitionsRDD[81] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_40 stored as values in memory (estimated size 21.0 KB, free 480.4 MB) 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_40_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.4 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_40_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:37 INFO SparkContext: Created broadcast 40 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:37 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 40 (MapPartitionsRDD[81] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:37 INFO YarnClusterScheduler: Adding task set 40.0 with 1 tasks 17/08/07 03:06:37 INFO TaskSetManager: Starting task 0.0 in stage 40.0 (TID 40, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_40_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO TaskSetManager: Finished task 0.0 in stage 40.0 (TID 40) in 22 ms on host4 (1/1) 17/08/07 03:06:37 INFO YarnClusterScheduler: Removed TaskSet 40.0, whose tasks have all completed, from pool 17/08/07 03:06:37 INFO DAGScheduler: ResultStage 40 (treeAggregate at LogisticRegression.scala:1018) finished in 0.022 s 17/08/07 03:06:37 INFO DAGScheduler: Job 35 finished: treeAggregate at LogisticRegression.scala:1018, took 0.026758 s 17/08/07 03:06:37 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:37 INFO LBFGS: Val and Grad Norm: 0.563125 (rel: 0.0433) 0.304538 17/08/07 03:06:37 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:37 INFO DAGScheduler: Got job 36 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:37 INFO DAGScheduler: Final stage: ResultStage 41 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:37 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:37 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:37 INFO DAGScheduler: Submitting ResultStage 41 (MapPartitionsRDD[82] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_41 stored as values in memory (estimated size 21.0 KB, free 480.3 MB) 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_41_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.3 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_41_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:37 INFO SparkContext: Created broadcast 41 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:37 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 41 (MapPartitionsRDD[82] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:37 INFO YarnClusterScheduler: Adding task set 41.0 with 1 tasks 17/08/07 03:06:37 INFO TaskSetManager: Starting task 0.0 in stage 41.0 (TID 41, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_41_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO TaskSetManager: Finished task 0.0 in stage 41.0 (TID 41) in 23 ms on host4 (1/1) 17/08/07 03:06:37 INFO YarnClusterScheduler: Removed TaskSet 41.0, whose tasks have all completed, from pool 17/08/07 03:06:37 INFO DAGScheduler: ResultStage 41 (treeAggregate at LogisticRegression.scala:1018) finished in 0.023 s 17/08/07 03:06:37 INFO DAGScheduler: Job 36 finished: treeAggregate at LogisticRegression.scala:1018, took 0.027919 s 17/08/07 03:06:37 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:37 INFO LBFGS: Val and Grad Norm: 0.551987 (rel: 0.0198) 0.00806565 17/08/07 03:06:37 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:37 INFO DAGScheduler: Got job 37 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:37 INFO DAGScheduler: Final stage: ResultStage 42 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:37 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:37 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:37 INFO DAGScheduler: Submitting ResultStage 42 (MapPartitionsRDD[83] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_42 stored as values in memory (estimated size 21.0 KB, free 480.3 MB) 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_42_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.3 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_42_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:37 INFO SparkContext: Created broadcast 42 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:37 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 42 (MapPartitionsRDD[83] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:37 INFO YarnClusterScheduler: Adding task set 42.0 with 1 tasks 17/08/07 03:06:37 INFO TaskSetManager: Starting task 0.0 in stage 42.0 (TID 42, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_42_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO TaskSetManager: Finished task 0.0 in stage 42.0 (TID 42) in 22 ms on host4 (1/1) 17/08/07 03:06:37 INFO YarnClusterScheduler: Removed TaskSet 42.0, whose tasks have all completed, from pool 17/08/07 03:06:37 INFO DAGScheduler: ResultStage 42 (treeAggregate at LogisticRegression.scala:1018) finished in 0.023 s 17/08/07 03:06:37 INFO DAGScheduler: Job 37 finished: treeAggregate at LogisticRegression.scala:1018, took 0.027802 s 17/08/07 03:06:37 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:37 INFO LBFGS: Val and Grad Norm: 0.551658 (rel: 0.000594) 0.0127115 17/08/07 03:06:37 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:37 INFO DAGScheduler: Got job 38 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:37 INFO DAGScheduler: Final stage: ResultStage 43 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:37 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:37 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:37 INFO DAGScheduler: Submitting ResultStage 43 (MapPartitionsRDD[84] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_43 stored as values in memory (estimated size 21.0 KB, free 480.3 MB) 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_43_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.3 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_43_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:37 INFO SparkContext: Created broadcast 43 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:37 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 43 (MapPartitionsRDD[84] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:37 INFO YarnClusterScheduler: Adding task set 43.0 with 1 tasks 17/08/07 03:06:37 INFO TaskSetManager: Starting task 0.0 in stage 43.0 (TID 43, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_43_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO TaskSetManager: Finished task 0.0 in stage 43.0 (TID 43) in 24 ms on host4 (1/1) 17/08/07 03:06:37 INFO YarnClusterScheduler: Removed TaskSet 43.0, whose tasks have all completed, from pool 17/08/07 03:06:37 INFO DAGScheduler: ResultStage 43 (treeAggregate at LogisticRegression.scala:1018) finished in 0.025 s 17/08/07 03:06:37 INFO DAGScheduler: Job 38 finished: treeAggregate at LogisticRegression.scala:1018, took 0.029595 s 17/08/07 03:06:37 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:37 INFO LBFGS: Val and Grad Norm: 0.551631 (rel: 5.01e-05) 0.00552583 17/08/07 03:06:37 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:37 INFO DAGScheduler: Got job 39 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:37 INFO DAGScheduler: Final stage: ResultStage 44 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:37 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:37 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:37 INFO DAGScheduler: Submitting ResultStage 44 (MapPartitionsRDD[85] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_44 stored as values in memory (estimated size 21.0 KB, free 480.3 MB) 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_44_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.3 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_44_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:37 INFO SparkContext: Created broadcast 44 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:37 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 44 (MapPartitionsRDD[85] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:37 INFO YarnClusterScheduler: Adding task set 44.0 with 1 tasks 17/08/07 03:06:37 INFO TaskSetManager: Starting task 0.0 in stage 44.0 (TID 44, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_44_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO TaskSetManager: Finished task 0.0 in stage 44.0 (TID 44) in 27 ms on host4 (1/1) 17/08/07 03:06:37 INFO YarnClusterScheduler: Removed TaskSet 44.0, whose tasks have all completed, from pool 17/08/07 03:06:37 INFO DAGScheduler: ResultStage 44 (treeAggregate at LogisticRegression.scala:1018) finished in 0.027 s 17/08/07 03:06:37 INFO DAGScheduler: Job 39 finished: treeAggregate at LogisticRegression.scala:1018, took 0.033463 s 17/08/07 03:06:37 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:37 INFO LBFGS: Val and Grad Norm: 0.551629 (rel: 4.02e-06) 0.000701662 17/08/07 03:06:37 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:37 INFO DAGScheduler: Got job 40 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:37 INFO DAGScheduler: Final stage: ResultStage 45 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:37 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:37 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:37 INFO DAGScheduler: Submitting ResultStage 45 (MapPartitionsRDD[86] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_45 stored as values in memory (estimated size 21.0 KB, free 480.2 MB) 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_45_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.2 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_45_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:37 INFO SparkContext: Created broadcast 45 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:37 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 45 (MapPartitionsRDD[86] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:37 INFO YarnClusterScheduler: Adding task set 45.0 with 1 tasks 17/08/07 03:06:37 INFO TaskSetManager: Starting task 0.0 in stage 45.0 (TID 45, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_45_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO TaskSetManager: Finished task 0.0 in stage 45.0 (TID 45) in 25 ms on host4 (1/1) 17/08/07 03:06:37 INFO YarnClusterScheduler: Removed TaskSet 45.0, whose tasks have all completed, from pool 17/08/07 03:06:37 INFO DAGScheduler: ResultStage 45 (treeAggregate at LogisticRegression.scala:1018) finished in 0.025 s 17/08/07 03:06:37 INFO DAGScheduler: Job 40 finished: treeAggregate at LogisticRegression.scala:1018, took 0.030466 s 17/08/07 03:06:37 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:37 INFO LBFGS: Val and Grad Norm: 0.551629 (rel: 8.95e-08) 7.80036e-06 17/08/07 03:06:37 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:37 INFO DAGScheduler: Got job 41 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:37 INFO DAGScheduler: Final stage: ResultStage 46 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:37 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:37 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:37 INFO DAGScheduler: Submitting ResultStage 46 (MapPartitionsRDD[87] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_46 stored as values in memory (estimated size 21.0 KB, free 480.2 MB) 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_46_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.2 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_46_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:37 INFO SparkContext: Created broadcast 46 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:37 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 46 (MapPartitionsRDD[87] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:37 INFO YarnClusterScheduler: Adding task set 46.0 with 1 tasks 17/08/07 03:06:37 INFO TaskSetManager: Starting task 0.0 in stage 46.0 (TID 46, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_46_piece0 in memory on host4:33596 (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO TaskSetManager: Finished task 0.0 in stage 46.0 (TID 46) in 20 ms on host4 (1/1) 17/08/07 03:06:37 INFO YarnClusterScheduler: Removed TaskSet 46.0, whose tasks have all completed, from pool 17/08/07 03:06:37 INFO DAGScheduler: ResultStage 46 (treeAggregate at LogisticRegression.scala:1018) finished in 0.021 s 17/08/07 03:06:37 INFO DAGScheduler: Job 41 finished: treeAggregate at LogisticRegression.scala:1018, took 0.027575 s 17/08/07 03:06:37 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:37 INFO LBFGS: Val and Grad Norm: 0.551629 (rel: 7.92e-10) 6.67461e-06 17/08/07 03:06:37 INFO SparkContext: Starting job: treeAggregate at LogisticRegression.scala:1018 17/08/07 03:06:37 INFO DAGScheduler: Got job 42 (treeAggregate at LogisticRegression.scala:1018) with 1 output partitions 17/08/07 03:06:37 INFO DAGScheduler: Final stage: ResultStage 47 (treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:37 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:37 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:37 INFO DAGScheduler: Submitting ResultStage 47 (MapPartitionsRDD[88] at treeAggregate at LogisticRegression.scala:1018), which has no missing parents 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_47 stored as values in memory (estimated size 21.0 KB, free 480.2 MB) 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_47_piece0 stored as bytes in memory (estimated size 9.0 KB, free 480.2 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_47_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:37 INFO SparkContext: Created broadcast 47 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:37 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 47 (MapPartitionsRDD[88] at treeAggregate at LogisticRegression.scala:1018) 17/08/07 03:06:37 INFO YarnClusterScheduler: Adding task set 47.0 with 1 tasks 17/08/07 03:06:37 INFO TaskSetManager: Starting task 0.0 in stage 47.0 (TID 47, host4, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_47_piece0 in memory on host4:33596 (size: 9.0 KB, free: 510.9 MB) 17/08/07 03:06:37 INFO TaskSetManager: Finished task 0.0 in stage 47.0 (TID 47) in 19 ms on host4 (1/1) 17/08/07 03:06:37 INFO YarnClusterScheduler: Removed TaskSet 47.0, whose tasks have all completed, from pool 17/08/07 03:06:37 INFO DAGScheduler: ResultStage 47 (treeAggregate at LogisticRegression.scala:1018) finished in 0.020 s 17/08/07 03:06:37 INFO DAGScheduler: Job 42 finished: treeAggregate at LogisticRegression.scala:1018, took 0.026215 s 17/08/07 03:06:37 INFO LBFGS: Step Size: 1.000 17/08/07 03:06:37 INFO LBFGS: Val and Grad Norm: 0.551629 (rel: 1.56e-12) 8.51268e-08 17/08/07 03:06:37 INFO MapPartitionsRDD: Removing RDD 52 from persistence list 17/08/07 03:06:37 INFO BlockManager: Removing RDD 52 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_47_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_47_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 81 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_46_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_46_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 80 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_45_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_45_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 79 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_44_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_44_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 78 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_43_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_43_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 77 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_42_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_42_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 76 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_41_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_41_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 75 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_40_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.6 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_40_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 74 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_39_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_39_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 73 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_38_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_38_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 72 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_37_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_37_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.0 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 71 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_36_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_36_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 70 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_35_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_35_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 69 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_34_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_34_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 68 17/08/07 03:06:37 INFO SparkContext: Starting job: dfToCols at NativeMethodAccessorImpl.java:-2 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_33_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:37 INFO DAGScheduler: Got job 43 (dfToCols at NativeMethodAccessorImpl.java:-2) with 1 output partitions 17/08/07 03:06:37 INFO DAGScheduler: Final stage: ResultStage 48 (dfToCols at NativeMethodAccessorImpl.java:-2) 17/08/07 03:06:37 INFO DAGScheduler: Parents of final stage: List() 17/08/07 03:06:37 INFO DAGScheduler: Missing parents: List() 17/08/07 03:06:37 INFO DAGScheduler: Submitting ResultStage 48 (MapPartitionsRDD[96] at dfToCols at NativeMethodAccessorImpl.java:-2), which has no missing parents 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_33_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_48 stored as values in memory (estimated size 21.3 KB, free 480.6 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 67 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_32_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_32_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 66 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_48_piece0 stored as bytes in memory (estimated size 9.4 KB, free 480.6 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_48_piece0 in memory on xxx.xx.xx.xxx:43180 (size: 9.4 KB, free: 480.7 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_31_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:37 INFO SparkContext: Created broadcast 48 from broadcast at DAGScheduler.scala:1008 17/08/07 03:06:37 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 48 (MapPartitionsRDD[96] at dfToCols at NativeMethodAccessorImpl.java:-2) 17/08/07 03:06:37 INFO YarnClusterScheduler: Adding task set 48.0 with 1 tasks 17/08/07 03:06:37 INFO TaskSetManager: Starting task 0.0 in stage 48.0 (TID 48, host6, partition 0,PROCESS_LOCAL, 16546 bytes) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_31_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 65 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_30_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_30_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 64 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_29_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_29_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 63 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_28_piece0 on xxx.xx.xx.xxx:43180 in memory (size: 9.0 KB, free: 480.7 MB) 17/08/07 03:06:37 INFO BlockManagerInfo: Removed broadcast_28_piece0 on host4:33596 in memory (size: 9.0 KB, free: 511.1 MB) 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 62 17/08/07 03:06:37 INFO BlockManager: Removing RDD 52 17/08/07 03:06:37 INFO ContextCleaner: Cleaned RDD 52 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 48 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 47 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 46 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 45 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 44 17/08/07 03:06:37 INFO ContextCleaner: Cleaned accumulator 43 17/08/07 03:06:37 INFO BlockManagerInfo: Added broadcast_48_piece0 in memory on host6:46860 (size: 9.4 KB, free: 511.1 MB) 17/08/07 03:06:37 INFO TaskSetManager: Finished task 0.0 in stage 48.0 (TID 48) in 168 ms on host6 (1/1) 17/08/07 03:06:37 INFO YarnClusterScheduler: Removed TaskSet 48.0, whose tasks have all completed, from pool 17/08/07 03:06:37 INFO DAGScheduler: ResultStage 48 (dfToCols at NativeMethodAccessorImpl.java:-2) finished in 0.169 s 17/08/07 03:06:37 INFO DAGScheduler: Job 43 finished: dfToCols at NativeMethodAccessorImpl.java:-2, took 0.176752 s 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static/sql,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/execution/json,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/execution,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/json,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null} 17/08/07 03:06:37 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null} 17/08/07 03:06:37 INFO SparkUI: Stopped Spark web UI at http://xxx.xx.xx.xxx:56049 17/08/07 03:06:37 INFO YarnAllocator: Driver requested a total number of 0 executor(s). 17/08/07 03:06:37 INFO YarnClusterSchedulerBackend: Shutting down all executors 17/08/07 03:06:37 INFO YarnAllocator: Canceling requests for 0 executor containers 17/08/07 03:06:37 WARN YarnAllocator: Expected to find pending requests, but found none. 17/08/07 03:06:37 INFO YarnClusterSchedulerBackend: Asking each executor to shut down 17/08/07 03:06:37 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices (serviceOption=None, services=List(), started=false) 17/08/07 03:06:37 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 17/08/07 03:06:37 INFO MemoryStore: MemoryStore cleared 17/08/07 03:06:37 INFO BlockManager: BlockManager stopped 17/08/07 03:06:37 INFO BlockManagerMaster: BlockManagerMaster stopped 17/08/07 03:06:37 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 17/08/07 03:06:37 INFO SparkContext: Successfully stopped SparkContext 17/08/07 03:06:37 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 17/08/07 03:06:37 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 17/08/07 03:06:37 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 17/08/07 03:06:38 INFO ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0 17/08/07 03:06:38 INFO ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED 17/08/07 03:06:38 INFO AMRMClientImpl: Waiting for application to be successfully unregistered. 17/08/07 03:06:39 INFO ApplicationMaster: Deleting staging directory .sparkStaging/application_1502070770869_0012 17/08/07 03:06:39 INFO ShutdownHookManager: Shutdown hook called 17/08/07 03:06:39 INFO ShutdownHookManager: Deleting directory /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1502070770869_0012/spark-f54d2131-c555-4210-b0b5-9843a3b12a38 End of LogType:stderr LogType:stdout Log Upload Time:Mon Aug 07 03:06:40 +0000 2017 LogLength:1201 Log Contents: Loading required package: methods Attaching package: ‘SparkR’ The following objects are masked from ‘package:stats’: cov, filter, lag, na.omit, predict, sd, var The following objects are masked from ‘package:base’: colnames, colnames<-, endsWith, intersect, rank, rbind, sample, startsWith, subset, summary, table, transform $devianceResiduals Min Max -1.307112 1.412532 $coefficients Estimate Std. Error t value Pr(>|t|) (Intercept) 2.251393 0.3697543 6.08889 9.568102e-09 Sepal_Width 0.8035609 0.106339 7.556598 4.187317e-12 Species_versicolor 1.458743 0.1121079 13.01195 0 Species_virginica 1.946817 0.100015 19.46525 0 Sepal_Length prediction 1 5.1 5.063856 2 4.9 4.662076 3 4.7 4.822788 4 4.6 4.742432 5 5.0 5.144212 6 5.4 5.385281 $coefficients Estimate (Intercept) -13.046005 Sepal_Length 1.902373 Sepal_Width 0.404655 Species prediction 1 versicolor 1 2 versicolor 1 3 versicolor 1 4 versicolor 0 5 versicolor 1 6 versicolor 0 End of LogType:stdout Container: container_e02_1502070770869_0012_01_000002 on host6_25454_1502075200479 LogAggregationType: AGGREGATED ============================================================================================================================ LogType:directory.info Log Upload Time:Mon Aug 07 03:06:40 +0000 2017 LogLength:12325 Log Contents: ls -l: total 12 -rw------- 1 hrt_qa hadoop 633 Aug 7 03:06 container_tokens -rwx------ 1 hrt_qa hadoop 7666 Aug 7 03:06 launch_container.sh lrwxrwxrwx 1 hrt_qa hadoop 60 Aug 7 03:06 ml.R -> /grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/17/ml.R lrwxrwxrwx 1 hrt_qa hadoop 93 Aug 7 03:06 __spark_conf__ -> /grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/18/__spark_conf__5875502619868816278.zip lrwxrwxrwx 1 hrt_qa hadoop 61 Aug 7 03:06 __spark__.jar -> /grid/0/hadoop/yarn/local/filecache/12/spark-hdp-assembly.jar lrwxrwxrwx 1 hrt_qa hadoop 66 Aug 7 03:06 sparkr -> /grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/16/sparkr.zip drwxr-s--- 2 hrt_qa hadoop 6 Aug 7 03:06 tmp find -L . -maxdepth 5 -ls: 538979418 0 drwxr-s--- 3 hrt_qa hadoop 130 Aug 7 03:06 . 1076578328 0 drwxr-s--- 2 hrt_qa hadoop 6 Aug 7 03:06 ./tmp 538979419 8 -rwx------ 1 hrt_qa hadoop 7666 Aug 7 03:06 ./launch_container.sh 538979420 4 -rw------- 1 hrt_qa hadoop 633 Aug 7 03:06 ./container_tokens 4564322423 0 drwx------ 3 hrt_qa hrt_qa 27 Aug 7 03:06 ./sparkr 4833784887 4 drwx------ 10 hrt_qa hrt_qa 4096 Aug 7 03:06 ./sparkr/SparkR 4833784888 8 -r-x------ 1 hrt_qa hrt_qa 7018 Aug 7 03:06 ./sparkr/SparkR/NAMESPACE 5104143379 0 drwx------ 2 hrt_qa hrt_qa 112 Aug 7 03:06 ./sparkr/SparkR/Meta 5104143380 4 -r-x------ 1 hrt_qa hrt_qa 164 Aug 7 03:06 ./sparkr/SparkR/Meta/Rd.rds 5104143381 4 -r-x------ 1 hrt_qa hrt_qa 188 Aug 7 03:06 ./sparkr/SparkR/Meta/hsearch.rds 5104143382 4 -r-x------ 1 hrt_qa hrt_qa 1717 Aug 7 03:06 ./sparkr/SparkR/Meta/nsInfo.rds 5104143383 4 -r-x------ 1 hrt_qa hrt_qa 746 Aug 7 03:06 ./sparkr/SparkR/Meta/package.rds 5104143384 4 -r-x------ 1 hrt_qa hrt_qa 38 Aug 7 03:06 ./sparkr/SparkR/Meta/links.rds 5369762929 0 drwx------ 2 hrt_qa hrt_qa 46 Aug 7 03:06 ./sparkr/SparkR/test_support 5369762930 4 -r-x------ 1 hrt_qa hrt_qa 2886 Aug 7 03:06 ./sparkr/SparkR/test_support/sparktestjar_2.10-1.0.jar 5641025574 0 drwx------ 2 hrt_qa hrt_qa 112 Aug 7 03:06 ./sparkr/SparkR/help 5641025575 4 -r-x------ 1 hrt_qa hrt_qa 64 Aug 7 03:06 ./sparkr/SparkR/help/paths.rds 5641025576 0 -r-x------ 1 hrt_qa hrt_qa 0 Aug 7 03:06 ./sparkr/SparkR/help/SparkR.rdb 5641025577 4 -r-x------ 1 hrt_qa hrt_qa 60 Aug 7 03:06 ./sparkr/SparkR/help/aliases.rds 5641025578 0 -r-x------ 1 hrt_qa hrt_qa 0 Aug 7 03:06 ./sparkr/SparkR/help/AnIndex 5641025579 4 -r-x------ 1 hrt_qa hrt_qa 113 Aug 7 03:06 ./sparkr/SparkR/help/SparkR.rdx 5905974365 0 drwx------ 3 hrt_qa hrt_qa 29 Aug 7 03:06 ./sparkr/SparkR/tests 6176352284 4 drwx------ 2 hrt_qa hrt_qa 4096 Aug 7 03:06 ./sparkr/SparkR/tests/testthat 6176352285 28 -r-x------ 1 hrt_qa hrt_qa 27490 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_rdd.R 6176352286 4 -r-x------ 1 hrt_qa hrt_qa 1957 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_client.R 6176352287 4 -r-x------ 1 hrt_qa hrt_qa 3981 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_context.R 6176352288 4 -r-x------ 1 hrt_qa hrt_qa 3691 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_binary_function.R 6176352289 4 -r-x------ 1 hrt_qa hrt_qa 3171 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_take.R 6176352290 8 -r-x------ 1 hrt_qa hrt_qa 5137 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_textFile.R 6176352291 8 -r-x------ 1 hrt_qa hrt_qa 4380 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_parallelize_collect.R 6176352292 8 -r-x------ 1 hrt_qa hrt_qa 4859 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_mllib.R 6176352293 4 -r-x------ 1 hrt_qa hrt_qa 1241 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/jarTest.R 6176352294 4 -r-x------ 1 hrt_qa hrt_qa 1542 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_includeJAR.R 6176352295 4 -r-x------ 1 hrt_qa hrt_qa 3064 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_binaryFile.R 6176352296 4 -r-x------ 1 hrt_qa hrt_qa 1611 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_broadcast.R 6176352297 4 -r-x------ 1 hrt_qa hrt_qa 1810 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_includePackage.R 6176352298 4 -r-x------ 1 hrt_qa hrt_qa 2327 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_Serde.R 6176352299 4 -r-x------ 1 hrt_qa hrt_qa 996 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/packageInAJarTest.R 6176352300 8 -r-x------ 1 hrt_qa hrt_qa 5201 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_utils.R 6176352301 8 -r-x------ 1 hrt_qa hrt_qa 7922 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_shuffle.R 6176352302 72 -r-x------ 1 hrt_qa hrt_qa 70877 Aug 7 03:06 ./sparkr/SparkR/tests/testthat/test_sparkSQL.R 4833784889 4 -r-x------ 1 hrt_qa hrt_qa 678 Aug 7 03:06 ./sparkr/SparkR/DESCRIPTION 6444090371 0 drwx------ 2 hrt_qa hrt_qa 49 Aug 7 03:06 ./sparkr/SparkR/html 6444090372 4 -r-x------ 1 hrt_qa hrt_qa 923 Aug 7 03:06 ./sparkr/SparkR/html/00Index.html 6444090373 4 -r-x------ 1 hrt_qa hrt_qa 1319 Aug 7 03:06 ./sparkr/SparkR/html/R.css 6711872535 0 drwx------ 2 hrt_qa hrt_qa 69 Aug 7 03:06 ./sparkr/SparkR/R 6711872536 4 -r-x------ 1 hrt_qa hrt_qa 1056 Aug 7 03:06 ./sparkr/SparkR/R/SparkR 6711872537 804 -r-x------ 1 hrt_qa hrt_qa 821006 Aug 7 03:06 ./sparkr/SparkR/R/SparkR.rdb 6711872538 20 -r-x------ 1 hrt_qa hrt_qa 19182 Aug 7 03:06 ./sparkr/SparkR/R/SparkR.rdx 6982347902 0 drwx------ 2 hrt_qa hrt_qa 48 Aug 7 03:06 ./sparkr/SparkR/profile 6982347903 4 -r-x------ 1 hrt_qa hrt_qa 1696 Aug 7 03:06 ./sparkr/SparkR/profile/shell.R 6982479872 4 -r-x------ 1 hrt_qa hrt_qa 958 Aug 7 03:06 ./sparkr/SparkR/profile/general.R 7250064386 0 drwx------ 2 hrt_qa hrt_qa 48 Aug 7 03:06 ./sparkr/SparkR/worker 7250064387 8 -r-x------ 1 hrt_qa hrt_qa 5884 Aug 7 03:06 ./sparkr/SparkR/worker/worker.R 7250064388 4 -r-x------ 1 hrt_qa hrt_qa 1863 Aug 7 03:06 ./sparkr/SparkR/worker/daemon.R 7519410181 4 -r-x------ 1 hrt_qa hrt_qa 1903 Aug 7 03:06 ./ml.R 7249122306 186408 -r-xr-xr-x 1 yarn hadoop 190878097 Aug 7 02:35 ./__spark__.jar 8324596740 4 drwx------ 2 hrt_qa hrt_qa 4096 Aug 7 03:06 ./__spark_conf__ 8324596741 4 -r-x------ 1 hrt_qa hrt_qa 620 Aug 7 03:06 ./__spark_conf__/log4j.properties 8324596742 8 -r-x------ 1 hrt_qa hrt_qa 6606 Aug 7 03:06 ./__spark_conf__/hadoop-env.sh 8324596743 8 -r-x------ 1 hrt_qa hrt_qa 7831 Aug 7 03:06 ./__spark_conf__/mapred-site.xml 8324596744 4 -r-x------ 1 hrt_qa hrt_qa 3085 Aug 7 03:06 ./__spark_conf__/hadoop-metrics2.properties 8324596745 28 -r-x------ 1 hrt_qa hrt_qa 24749 Aug 7 03:06 ./__spark_conf__/yarn-site.xml 8324596746 4 -r-x------ 1 hrt_qa hrt_qa 3979 Aug 7 03:06 ./__spark_conf__/hadoop-env.cmd 8324596747 8 -r-x------ 1 hrt_qa hrt_qa 7596 Aug 7 03:06 ./__spark_conf__/core-site.xml 8324596748 0 -r-x------ 1 hrt_qa hrt_qa 0 Aug 7 03:06 ./__spark_conf__/yarn.exclude 8324596749 4 -r-x------ 1 hrt_qa hrt_qa 1 Aug 7 03:06 ./__spark_conf__/dfs.exclude 8324596750 4 -r-x------ 1 hrt_qa hrt_qa 1631 Aug 7 03:06 ./__spark_conf__/kms-log4j.properties 8324596751 4 -r-x------ 1 hrt_qa hrt_qa 2250 Aug 7 03:06 ./__spark_conf__/yarn-env.cmd 8324596752 4 -r-x------ 1 hrt_qa hrt_qa 917 Aug 7 03:06 ./__spark_conf__/ssl-client.xml 8324596753 4 -r-x------ 1 hrt_qa hrt_qa 2313 Aug 7 03:06 ./__spark_conf__/capacity-scheduler.xml 8324596754 4 -r-x------ 1 hrt_qa hrt_qa 2610 Aug 7 03:06 ./__spark_conf__/ranger-hdfs-audit.xml 8324596755 4 -r-x------ 1 hrt_qa hrt_qa 3518 Aug 7 03:06 ./__spark_conf__/kms-acls.xml 8324596756 4 -r-x------ 1 hrt_qa hrt_qa 758 Aug 7 03:06 ./__spark_conf__/mapred-site.xml.template 8324596757 4 -r-x------ 1 hrt_qa hrt_qa 2358 Aug 7 03:06 ./__spark_conf__/topology_script.py 8324596758 4 -r-x------ 1 hrt_qa hrt_qa 333 Aug 7 03:06 ./__spark_conf__/yarn_nm_jaas.conf 8324596759 4 -r-x------ 1 hrt_qa hrt_qa 1335 Aug 7 03:06 ./__spark_conf__/configuration.xsl 8324596760 4 -r-x------ 1 hrt_qa hrt_qa 64 Aug 7 03:06 ./__spark_conf__/ranger-security.xml 8324596761 8 -r-x------ 1 hrt_qa hrt_qa 5532 Aug 7 03:06 ./__spark_conf__/yarn-env.sh 8324596762 12 -r-x------ 1 hrt_qa hrt_qa 11405 Aug 7 03:06 ./__spark_conf__/hdfs-site.xml 8324596763 4 -r-x------ 1 hrt_qa hrt_qa 337 Aug 7 03:06 ./__spark_conf__/yarn_ats_jaas.conf 8324596764 4 -r-x------ 1 hrt_qa hrt_qa 1020 Aug 7 03:06 ./__spark_conf__/commons-logging.properties 8324596765 4 -r-x------ 1 hrt_qa hrt_qa 333 Aug 7 03:06 ./__spark_conf__/hdfs_nn_jaas.conf 8324596766 4 -r-x------ 1 hrt_qa hrt_qa 333 Aug 7 03:06 ./__spark_conf__/hdfs_dn_jaas.conf 8324596767 4 -r-x------ 1 hrt_qa hrt_qa 1033 Aug 7 03:06 ./__spark_conf__/container-executor.cfg 8324596768 4 -r-x------ 1 hrt_qa hrt_qa 571 Aug 7 03:06 ./__spark_conf__/yarn_jaas.conf 8324596769 8 -r-x------ 1 hrt_qa hrt_qa 4221 Aug 7 03:06 ./__spark_conf__/task-log4j.properties 8324596770 4 -r-x------ 1 hrt_qa hrt_qa 2490 Aug 7 03:06 ./__spark_conf__/hadoop-metrics.properties 8324596771 4 -r-x------ 1 hrt_qa hrt_qa 984 Aug 7 03:06 ./__spark_conf__/mapred-env.sh 8324596772 4 -r-x------ 1 hrt_qa hrt_qa 1602 Aug 7 03:06 ./__spark_conf__/health_check 8324596773 4 -r-x------ 1 hrt_qa hrt_qa 1001 Aug 7 03:06 ./__spark_conf__/ranger-policymgr-ssl-yarn.xml 8324596774 4 -r-x------ 1 hrt_qa hrt_qa 320 Aug 7 03:06 ./__spark_conf__/mapred_jaas.conf 8324596775 4 -r-x------ 1 hrt_qa hrt_qa 2316 Aug 7 03:06 ./__spark_conf__/ssl-client.xml.example 8324596776 4 -r-x------ 1 hrt_qa hrt_qa 1527 Aug 7 03:06 ./__spark_conf__/kms-env.sh 8324596777 4 -r-x------ 1 hrt_qa hrt_qa 1308 Aug 7 03:06 ./__spark_conf__/hadoop-policy.xml 8324596778 4 -r-x------ 1 hrt_qa hrt_qa 2610 Aug 7 03:06 ./__spark_conf__/ranger-yarn-audit.xml 8324596779 4 -r-x------ 1 hrt_qa hrt_qa 145 Aug 7 03:06 ./__spark_conf__/slaves 8324596780 4 -r-x------ 1 hrt_qa hrt_qa 287 Aug 7 03:06 ./__spark_conf__/topology_mappings.data 8324596781 4 -r-x------ 1 hrt_qa hrt_qa 1049 Aug 7 03:06 ./__spark_conf__/ranger-hdfs-security.xml 8324596782 8 -r-x------ 1 hrt_qa hrt_qa 4956 Aug 7 03:06 ./__spark_conf__/metrics.properties 8324596783 4 -r-x------ 1 hrt_qa hrt_qa 951 Aug 7 03:06 ./__spark_conf__/mapred-env.cmd 8324596784 4 -r-x------ 1 hrt_qa hrt_qa 1054 Aug 7 03:06 ./__spark_conf__/ssl-server.xml 8324596785 4 -r-x------ 1 hrt_qa hrt_qa 333 Aug 7 03:06 ./__spark_conf__/hdfs_jn_jaas.conf 8324596786 4 -r-x------ 1 hrt_qa hrt_qa 2697 Aug 7 03:06 ./__spark_conf__/ssl-server.xml.example 8324596787 4 -r-x------ 1 hrt_qa hrt_qa 995 Aug 7 03:06 ./__spark_conf__/ranger-policymgr-ssl.xml 8324596788 4 -r-x------ 1 hrt_qa hrt_qa 935 Aug 7 03:06 ./__spark_conf__/ranger-yarn-security.xml 8324596789 4 -r-x------ 1 hrt_qa hrt_qa 946 Aug 7 03:06 ./__spark_conf__/taskcontroller.cfg 8324596790 8 -r-x------ 1 hrt_qa hrt_qa 4113 Aug 7 03:06 ./__spark_conf__/mapred-queues.xml.template 8324596791 8 -r-x------ 1 hrt_qa hrt_qa 5511 Aug 7 03:06 ./__spark_conf__/kms-site.xml 8324596792 4 -r-x------ 1 hrt_qa hrt_qa 1661 Aug 7 03:06 ./__spark_conf__/__spark_conf__.properties broken symlinks(find -L . -maxdepth 5 -type l -ls): End of LogType:directory.info LogType:launch_container.sh Log Upload Time:Mon Aug 07 03:06:40 +0000 2017 LogLength:7666 Log Contents: #!/bin/bash export SPARK_YARN_STAGING_DIR=".sparkStaging/application_1502070770869_0012" export PATH="/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/var/lib/ambari-agent" export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/hdp/current/hadoop-client/conf"} export JAVA_HOME=${JAVA_HOME:-"/usr/lib/jvm/java-openjdk"} export SPARK_YARN_CACHE_FILES="hdfs://mycluster/hdp/apps/2.6.2.0-154/spark/spark-hdp-assembly.jar#__spark__.jar,hdfs://mycluster/user/hrt_qa/.sparkStaging/application_1502070770869_0012/ml.R#ml.R" export LANG="en_US.UTF-8" export SPARK_LOG_URL_STDOUT="https://host6:54321/node/containerlogs/container_e02_1502070770869_0012_01_000002/hrt_qa/stdout?start=-4096" export NM_HOST="host6" export SPARK_YARN_CACHE_FILES_FILE_SIZES="190878097,1903" export SPARK_YARN_CACHE_ARCHIVES_TIME_STAMPS="1502075165966,1502075166070" export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/lib64/R/lib:/usr/lib/jvm/jre/lib/amd64/server:/usr/lib/jvm/jre/lib/amd64:/usr/lib/jvm/java/lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib:/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:" export LOGNAME="hrt_qa" export JVM_PID="$$" export PWD="/grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1502070770869_0012/container_e02_1502070770869_0012_01_000002" export LOCAL_DIRS="/grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1502070770869_0012" export NM_HTTP_PORT="54321" export LOG_DIRS="/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000002" export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= " export NM_PORT="25454" export SPARK_YARN_CACHE_FILES_TIME_STAMPS="1502064724925,1502075165821" export USER="hrt_qa" export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/hdp/current/hadoop-yarn-nodemanager"} export CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark__.jar:/etc/hadoop/conf:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:/usr/hdp/current/ext/hadoop/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/2.6.2.0-154/hadoop/lib/hadoop-lzo-0.6.0.2.6.2.0-154.jar:/etc/hadoop/conf/secure:/usr/hdp/current/ext/hadoop/*" export SPARK_YARN_CACHE_ARCHIVES="hdfs://mycluster/user/hrt_qa/.sparkStaging/application_1502070770869_0012/sparkr.zip#sparkr,hdfs://mycluster/user/hrt_qa/.sparkStaging/application_1502070770869_0012/__spark_conf__5875502619868816278.zip#__spark_conf__" export SPARK_YARN_CACHE_ARCHIVES_FILE_SIZES="555499,144034" export SPARK_YARN_MODE="true" export SPARK_YARN_CACHE_FILES_VISIBILITIES="PUBLIC,PRIVATE" export HADOOP_TOKEN_FILE_LOCATION="/grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1502070770869_0012/container_e02_1502070770869_0012_01_000002/container_tokens" export NM_AUX_SERVICE_spark_shuffle="" export SPARK_USER="hrt_qa" export LOCAL_USER_DIRS="/grid/0/hadoop/yarn/local/usercache/hrt_qa/" export HADOOP_HOME="/usr/hdp/2.6.2.0-154/hadoop" export SPARK_LOG_URL_STDERR="https://host6:54321/node/containerlogs/container_e02_1502070770869_0012_01_000002/hrt_qa/stderr?start=-4096" export SPARK_YARN_CACHE_ARCHIVES_VISIBILITIES="PRIVATE,PRIVATE" export HOME="/home/" export NM_AUX_SERVICE_spark2_shuffle="" export CONTAINER_ID="container_e02_1502070770869_0012_01_000002" export MALLOC_ARENA_MAX="4" ln -sf "/grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/16/sparkr.zip" "sparkr" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/17/ml.R" "ml.R" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/grid/0/hadoop/yarn/local/filecache/12/spark-hdp-assembly.jar" "__spark__.jar" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/grid/0/hadoop/yarn/local/usercache/hrt_qa/filecache/18/__spark_conf__5875502619868816278.zip" "__spark_conf__" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi # Creating copy of launch script cp "launch_container.sh" "/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000002/launch_container.sh" chmod 640 "/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000002/launch_container.sh" # Determining directory contents echo "ls -l:" 1>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000002/directory.info" ls -l 1>>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000002/directory.info" echo "find -L . -maxdepth 5 -ls:" 1>>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000002/directory.info" find -L . -maxdepth 5 -ls 1>>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000002/directory.info" echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000002/directory.info" find -L . -maxdepth 5 -type l -ls 1>>"/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000002/directory.info" exec /bin/bash -c "LD_LIBRARY_PATH="/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms1024m -Xmx1024m -Djava.io.tmpdir=$PWD/tmp '-Dspark.authenticate=true' '-Dspark.ssl.trustStore=/etc/security/clientKeys/spark_all.jks' '-Dspark.ssl.keyStore=/etc/security/serverKeys/spark_keystore.jks' '-Dspark.authenticate.enableSaslEncryption=true' '-Dspark.driver.port=53417' '-Dspark.ssl.enabledAlgorithms=TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA' '-Dspark.ssl.enabled=true' '-Dspark.history.ui.port=18080' '-Dspark.ssl.trustStorePassword=clientTrustStorePassword' '-Dspark.ui.port=0' '-Dspark.ssl.protocol=TLS' '-Dspark.ssl.keyPassword=serverKeyPassPassword' '-Dspark.ssl.keyStorePassword=serverStorePassPassword' -Dspark.yarn.app.container.log.dir=/grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000002 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@xxx.xx.xx.xxx:53417 --executor-id 1 --hostname host6 --cores 1 --app-id application_1502070770869_0012 --user-class-path file:$PWD/__app__.jar 1> /grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000002/stdout 2> /grid/0/hadoop/yarn/log/application_1502070770869_0012/container_e02_1502070770869_0012_01_000002/stderr" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi End of LogType:launch_container.sh LogType:stderr Log Upload Time:Mon Aug 07 03:06:40 +0000 2017 LogLength:9099 Log Contents: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/grid/0/hadoop/yarn/local/filecache/12/spark-hdp-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.2.0-154/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 17/08/07 03:06:20 INFO CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT] 17/08/07 03:06:21 INFO SecurityManager: Changing view acls to: hrt_qa 17/08/07 03:06:21 INFO SecurityManager: Changing modify acls to: hrt_qa 17/08/07 03:06:21 INFO SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: Set(hrt_qa); users with modify permissions: Set(hrt_qa) 17/08/07 03:06:22 INFO SecurityManager: Changing view acls to: hrt_qa 17/08/07 03:06:22 INFO SecurityManager: Changing modify acls to: hrt_qa 17/08/07 03:06:22 INFO SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: Set(hrt_qa); users with modify permissions: Set(hrt_qa) 17/08/07 03:06:22 INFO Slf4jLogger: Slf4jLogger started 17/08/07 03:06:22 INFO Remoting: Starting remoting 17/08/07 03:06:22 INFO Remoting: Remoting started; listening on addresses :[akka.ssl.tcp://sparkExecutorActorSystem@host6:57725] 17/08/07 03:06:22 INFO Utils: Successfully started service 'sparkExecutorActorSystem' on port 57725. 17/08/07 03:06:22 INFO DiskBlockManager: Created local directory at /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1502070770869_0012/blockmgr-fb2e90de-0a38-482d-a3bb-3350066dd752 17/08/07 03:06:22 INFO MemoryStore: MemoryStore started with capacity 511.1 MB 17/08/07 03:06:22 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@xxx.xx.xx.xxx:53417 17/08/07 03:06:23 INFO CoarseGrainedExecutorBackend: Successfully registered with driver 17/08/07 03:06:23 INFO Executor: Starting executor ID 1 on host host6 17/08/07 03:06:23 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46860. 17/08/07 03:06:23 INFO NettyBlockTransferService: Server created on 46860 17/08/07 03:06:23 INFO BlockManagerMaster: Trying to register BlockManager 17/08/07 03:06:23 INFO BlockManagerMaster: Registered BlockManager 17/08/07 03:06:24 INFO CoarseGrainedExecutorBackend: Got assigned task 0 17/08/07 03:06:24 INFO Executor: Running task 0.0 in stage 0.0 (TID 0) 17/08/07 03:06:24 INFO TorrentBroadcast: Started reading broadcast variable 0 17/08/07 03:06:24 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 890.0 B, free 511.1 MB) 17/08/07 03:06:24 INFO TorrentBroadcast: Reading broadcast variable 0 took 157 ms 17/08/07 03:06:24 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1344.0 B, free 511.1 MB) 17/08/07 03:06:24 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 15502 bytes result sent to driver 17/08/07 03:06:30 INFO CoarseGrainedExecutorBackend: Got assigned task 5 17/08/07 03:06:30 INFO Executor: Running task 0.0 in stage 5.0 (TID 5) 17/08/07 03:06:30 INFO MapOutputTrackerWorker: Updating epoch to 2 and clearing cache 17/08/07 03:06:30 INFO TorrentBroadcast: Started reading broadcast variable 5 17/08/07 03:06:30 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 9.0 KB, free 511.1 MB) 17/08/07 03:06:30 INFO TorrentBroadcast: Reading broadcast variable 5 took 22 ms 17/08/07 03:06:30 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 20.7 KB, free 511.1 MB) 17/08/07 03:06:32 INFO GenerateUnsafeProjection: Code generated in 173.867777 ms 17/08/07 03:06:32 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS 17/08/07 03:06:32 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS 17/08/07 03:06:32 INFO RRDD: Times: boot = 0.856 s, init = 0.006 s, broadcast = 0.000 s, read-input = 0.001 s, compute = 0.001 s, write-output = 0.065 s, total = 0.929 s 17/08/07 03:06:32 INFO Executor: Finished task 0.0 in stage 5.0 (TID 5). 1628 bytes result sent to driver 17/08/07 03:06:32 INFO CoarseGrainedExecutorBackend: Got assigned task 7 17/08/07 03:06:32 INFO Executor: Running task 0.0 in stage 7.0 (TID 7) 17/08/07 03:06:32 INFO TorrentBroadcast: Started reading broadcast variable 7 17/08/07 03:06:33 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 9.9 KB, free 511.1 MB) 17/08/07 03:06:33 INFO TorrentBroadcast: Reading broadcast variable 7 took 22 ms 17/08/07 03:06:33 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 22.4 KB, free 511.1 MB) 17/08/07 03:06:33 INFO GenerateUnsafeProjection: Code generated in 46.453558 ms 17/08/07 03:06:33 INFO RRDD: Times: boot = 0.008 s, init = 0.003 s, broadcast = 0.000 s, read-input = 0.001 s, compute = 0.001 s, write-output = 0.062 s, total = 0.075 s 17/08/07 03:06:33 INFO Executor: Finished task 0.0 in stage 7.0 (TID 7). 1138 bytes result sent to driver 17/08/07 03:06:34 INFO CoarseGrainedExecutorBackend: Got assigned task 13 17/08/07 03:06:34 INFO Executor: Running task 0.0 in stage 13.0 (TID 13) 17/08/07 03:06:34 INFO MapOutputTrackerWorker: Updating epoch to 4 and clearing cache 17/08/07 03:06:34 INFO TorrentBroadcast: Started reading broadcast variable 13 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 6.4 KB, free 511.1 MB) 17/08/07 03:06:34 INFO TorrentBroadcast: Reading broadcast variable 13 took 16 ms 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 15.5 KB, free 511.1 MB) 17/08/07 03:06:34 INFO GeneratePredicate: Code generated in 6.379311 ms 17/08/07 03:06:34 INFO GenerateUnsafeProjection: Code generated in 7.2588 ms 17/08/07 03:06:34 INFO RRDD: Times: boot = 0.009 s, init = 0.004 s, broadcast = 0.000 s, read-input = 0.000 s, compute = 0.002 s, write-output = 0.073 s, total = 0.088 s 17/08/07 03:06:34 INFO Executor: Finished task 0.0 in stage 13.0 (TID 13). 1512 bytes result sent to driver 17/08/07 03:06:34 INFO CoarseGrainedExecutorBackend: Got assigned task 14 17/08/07 03:06:34 INFO Executor: Running task 0.0 in stage 14.0 (TID 14) 17/08/07 03:06:34 INFO MapOutputTrackerWorker: Updating epoch to 5 and clearing cache 17/08/07 03:06:34 INFO TorrentBroadcast: Started reading broadcast variable 14 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 1607.0 B, free 511.1 MB) 17/08/07 03:06:34 INFO TorrentBroadcast: Reading broadcast variable 14 took 12 ms 17/08/07 03:06:34 INFO MemoryStore: Block broadcast_14 stored as values in memory (estimated size 2.7 KB, free 511.1 MB) 17/08/07 03:06:34 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 4, fetching them 17/08/07 03:06:34 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@xxx.xx.xx.xxx:53417) 17/08/07 03:06:34 INFO MapOutputTrackerWorker: Got the output locations 17/08/07 03:06:34 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks 17/08/07 03:06:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 4 ms 17/08/07 03:06:34 INFO Executor: Finished task 0.0 in stage 14.0 (TID 14). 1378 bytes result sent to driver 17/08/07 03:06:37 INFO BlockManager: Removing RDD 52 17/08/07 03:06:37 INFO CoarseGrainedExecutorBackend: Got assigned task 48 17/08/07 03:06:37 INFO Executor: Running task 0.0 in stage 48.0 (TID 48) 17/08/07 03:06:37 INFO TorrentBroadcast: Started reading broadcast variable 48 17/08/07 03:06:37 INFO BlockManager: Removing RDD 52 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_48_piece0 stored as bytes in memory (estimated size 9.4 KB, free 511.1 MB) 17/08/07 03:06:37 INFO TorrentBroadcast: Reading broadcast variable 48 took 16 ms 17/08/07 03:06:37 INFO MemoryStore: Block broadcast_48 stored as values in memory (estimated size 21.3 KB, free 511.1 MB) 17/08/07 03:06:37 INFO GenerateUnsafeProjection: Code generated in 27.350661 ms 17/08/07 03:06:37 INFO GenerateSafeProjection: Code generated in 7.822978 ms 17/08/07 03:06:37 INFO Executor: Finished task 0.0 in stage 48.0 (TID 48). 1887 bytes result sent to driver 17/08/07 03:06:37 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown 17/08/07 03:06:37 INFO MemoryStore: MemoryStore cleared 17/08/07 03:06:37 INFO BlockManager: BlockManager stopped 17/08/07 03:06:37 INFO CoarseGrainedExecutorBackend: Driver from host4:53417 disconnected during shutdown 17/08/07 03:06:37 INFO CoarseGrainedExecutorBackend: Driver from xxx.xx.xx.xxx:53417 disconnected during shutdown 17/08/07 03:06:37 INFO ShutdownHookManager: Shutdown hook called End of LogType:stderr LogType:stdout Log Upload Time:Mon Aug 07 03:06:40 +0000 2017 LogLength:0 Log Contents: End of LogType:stdout