[AMBARI-10449] NameNode cannot to start on cluster installed via blueprint with NN HA enabled - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Critical
Resolution: Fixed
Affects Version/s: 2.0.0
Fix Version/s: 2.0.1
Component/s: ambari-server
Labels:
None

Description

Occasionally, NN HA enabled deploys through blueprint fails while starting the active NN. Based on the log, this is what is happening:

Pre-conditions:

Standby NN is the first NN in the list for dfsadmin commands. Some dfsadmin commands loop through list of NNs and fail-fast upon first communication failure

Repro steps

Start command for the two NNs begin - fairly simultaneously
Standby NN tries to bootstrap itself and thus tries to communicate with the active NN
- Standby continues trying till the active NN can respond and after bootstrap is successful standby NN goes down
Meanwhile, active NN is up and tries to find out if Safemode is ON - at this point standby NN is not up yet
- The dfsadmin call fails. The grep logic results in a false positive
- Due to false positive, attempt is made by the active NN to exit safemode which also fails
Standby NN starts but the damage is done - START command has already failed for active NN
This results in Ambari aborting all follow-up commands

Why does it happen occasionally? I think if the first host (alphabetically ordered) is the standby host then the dfsadmin commands fail due to the above theory.

Logs:

Active NN

2015-04-10 02:31:49,381 - call['hadoop dfsadmin -fs hdfs://mycluster -safemode get | grep 'Safe mode is OFF''] {}
2015-04-10 02:31:52,911 - call returned (1, 'DEPRECATED: Use of this script to execute hdfs command is deprecated.\r\nInstead use the hdfs command for it.\r\n\r\nSLF4J: Class path contains multiple SLF4J bindings.\r\nSLF4J: Found binding in [jar:file:/grid/0/hdp/2.3.0.0-1551/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]\r\nSLF4J: Found binding in [jar:file:/grid/0/hdp/2.3.0.0-1551/tez/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]\r\nSLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.\r\nSLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]\r\nsafemode: Call From devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125 to devdeploy-os-amb-r6-secha-1428629670-2.novalocal:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused\r')
2015-04-10 02:31:52,913 - Execute['hdfs --config /etc/hadoop/conf dfsadmin -fs hdfs://mycluster -safemode leave'] {'path': ['/usr/hdp/current/hadoop-client/bin'], 'user': 'hdfs'}

Standby NN

2015-04-10 02:31:29,065 - Try 1 out of 50
2015-04-10 02:31:29,066 - call['hdfs namenode -bootstrapStandby -nonInteractive'] {'logoutput': False, 'user': 'hdfs'}
2015-04-10 02:31:52,827 - call returned (0, "SLF4J: Class path contains multiple SLF4J bindings.\r\nSLF4J: Found binding in [jar:file:/grid/0/hdp/2.3.0.0-1551/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]\r\nSLF4J: Found binding in [jar:file:/grid/0/hdp/2.3.0.0-1551/tez/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]\r\nSLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.\r\nSLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]\r\n15/04/10 02:31:30 INFO namenode.NameNode: STARTUP_MSG: \r\n/************************************************************\r\nSTARTUP_MSG: Starting NameNode\r\nSTARTUP_MSG:   host = devdeploy-os-amb-r6-secha-1428629670-2.novalocal/192.168.75.130\r\nSTARTUP_MSG:   args = [-bootstrapStandby, -nonInteractive]\r\nSTARTUP_MSG:   version = 2.7.0.2.3.0.0-1551\r\nSTARTUP_MSG:   classpath = /usr/hdp/2.3.0.0-1551/hadoop/conf:/usr/hdp/2.3.0.0-1551/hadoop/lib/commons-configuration-1.6.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/httpmime-4.2.5.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/ranger-plugins-common-0.5.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/httpcore-4.2.5.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/commons-collections-3.2.1.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/javax.persistence-2.1.0.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/gson-2.2.4.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/commons-logging-1.1.3.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jetty-6.1.26.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/ranger-plugins-cred-0.5.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/activation-1.1.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/mockito-all-1.8.5.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/junit-4.11.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/zookeeper-3.4.6.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jsr305-3.0.0.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/hamcrest-core-1.3.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/commons-cli-1.2.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/avro-1.7.4.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/commons-digester-1.8.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/commons-compress-1.4.1.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jersey-json-1.9.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/noggit-0.6.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/commons-math3-3.1.1.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/ranger-plugins-audit-0.5.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/ranger_solrj-0.5.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/eclipselink-2.5.2-M1.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/commons-codec-1.4.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/ranger-hdfs-plugin-0.5.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/ojdbc6.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/curator-client-2.7.1.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jetty-util-6.1.26.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/guava-11.0.2.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/xz-1.0.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/commons-lang-2.6.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/httpclient-4.2.5.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jsch-0.1.42.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/curator-framework-2.7.1.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/stax-api-1.0-2.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/asm-3.2.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/commons-httpclient-3.1.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.3.0.0-1551/hadoop/lib/netty-3.6.2.Final.jar:/usr/hdp/2.3.0.0-1551/hadoop/.//hadoop-common-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop/.//hadoop-common-2.7.0.2.3.0.0-1551-tests.jar:/usr/hdp/2.3.0.0-1551/hadoop/.//hadoop-nfs.jar:/usr/hdp/2.3.0.0-1551/hadoop/.//hadoop-auth-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop/.//hadoop-annotations.jar:/usr/hdp/2.3.0.0-1551/hadoop/.//hadoop-common.jar:/usr/hdp/2.3.0.0-1551/hadoop/.//hadoop-common-tests.jar:/usr/hdp/2.3.0.0-1551/hadoop/.//hadoop-annotations-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop/.//hadoop-auth.jar:/usr/hdp/2.3.0.0-1551/hadoop/.//hadoop-nfs-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/./:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/jetty-6.1.26.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/jetty-util-6.1.26.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/asm-3.2.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/.//hadoop-hdfs-2.7.0.2.3.0.0-1551-tests.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/.//hadoop-hdfs-nfs.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/.//hadoop-hdfs-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/.//hadoop-hdfs-tests.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/hdp/2.3.0.0-1551/hadoop-hdfs/.//hadoop-hdfs-nfs-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/jettison-1.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/commons-collections-3.2.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/jetty-6.1.26.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/javax.inject-1.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/activation-1.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/zookeeper-3.4.6.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/commons-io-2.4.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/guice-3.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/jetty-util-6.1.26.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/zookeeper-3.4.6.2.3.0.0-1551-tests.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/guava-11.0.2.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/xz-1.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/asm-3.2.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-client.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-registry.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-server-tests.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-client-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-registry-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-api-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-server-tests-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-server-common-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-common-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/2.3.0.0-1551/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/junit-4.11.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/guice-3.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/xz-1.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/asm-3.2.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-extras.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//commons-configuration-1.6.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-archives-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//httpcore-4.2.5.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-client-core-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jettison-1.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//commons-collections-3.2.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-datajoin.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jackson-jaxrs-1.9.13.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//gson-2.2.4.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//commons-logging-1.1.3.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-gridmix.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jetty-6.1.26.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-aws-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-rumen.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//activation-1.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-client-core.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-archives.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//htrace-core-3.1.0-incubating.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//mockito-all-1.8.5.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-extras-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//junit-4.11.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-auth-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//zookeeper-3.4.6.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jackson-databind-2.2.3.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jsr305-3.0.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hamcrest-core-1.3.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-aws.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-streaming.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-streaming-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jackson-core-asl-1.9.13.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jersey-server-1.9.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//commons-cli-1.2.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//avro-1.7.4.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//commons-digester-1.8.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-distcp.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-tests.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-sls.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//commons-compress-1.4.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-datajoin-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//commons-beanutils-core-1.8.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jersey-json-1.9.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-ant.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jets3t-0.9.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-openstack.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//commons-math3-3.1.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//log4j-1.2.17.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jersey-core-1.9.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-client-common-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//commons-io-2.4.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//commons-net-3.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//xmlenc-0.52.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-sls-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//snappy-java-1.0.4.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-examples.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-client-common.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//paranamer-2.3.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//api-util-1.0.0-M20.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-examples-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-ant-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//commons-codec-1.4.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//curator-recipes-2.7.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-client-app.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jackson-core-2.2.3.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//curator-client-2.7.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-gridmix-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//protobuf-java-2.5.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jetty-util-6.1.26.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jackson-xc-1.9.13.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.0.2.3.0.0-1551-tests.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-rumen-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//commons-beanutils-1.7.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//java-xmlbuilder-0.4.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-distcp-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//guava-11.0.2.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//xz-1.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//joda-time-2.7.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jaxb-api-2.2.2.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//commons-lang-2.6.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//aws-java-sdk-1.7.4.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//httpclient-4.2.5.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-auth.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//servlet-api-2.5.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-client-app-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jackson-annotations-2.2.3.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jsch-0.1.42.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//curator-framework-2.7.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-azure-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//stax-api-1.0-2.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//asm-3.2.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jaxb-impl-2.2.3-1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//commons-lang3-3.3.2.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//commons-httpclient-3.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//jsp-api-2.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//metrics-core-3.0.1.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-azure.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//netty-3.6.2.Final.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//azure-storage-2.0.0.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-openstack-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/hadoop-mapreduce/.//hadoop-mapreduce-client-hs.jar::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java.jar:/usr/share/java/mysql-jdbc-qe.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-extras.jar:/usr/hdp/current/hadoop-mapreduce-client/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-archives-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/httpcore-4.2.5.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/jettison-1.1.jar:/usr/hdp/current/hadoop-mapreduce-client/commons-collections-3.2.1.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-datajoin.jar:/usr/hdp/current/hadoop-mapreduce-client/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/gson-2.2.4.jar:/usr/hdp/current/hadoop-mapreduce-client/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-gridmix.jar:/usr/hdp/current/hadoop-mapreduce-client/jetty-6.1.26.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-aws-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-rumen.jar:/usr/hdp/current/hadoop-mapreduce-client/activation-1.1.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-archives.jar:/usr/hdp/current/hadoop-mapreduce-client/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-mapreduce-client/mockito-all-1.8.5.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-extras-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/junit-4.11.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-auth-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/zookeeper-3.4.6.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-mapreduce-client/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-mapreduce-client/hamcrest-core-1.3.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-aws.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-mapreduce-client/jersey-server-1.9.jar:/usr/hdp/current/hadoop-mapreduce-client/commons-cli-1.2.jar:/usr/hdp/current/hadoop-mapreduce-client/avro-1.7.4.jar:/usr/hdp/current/hadoop-mapreduce-client/commons-digester-1.8.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-distcp.jar:/usr/hdp/current/hadoop-mapreduce-client/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-tests.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-sls.jar:/usr/hdp/current/hadoop-mapreduce-client/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-datajoin-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-mapreduce-client/jersey-json-1.9.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-ant.jar:/usr/hdp/current/hadoop-mapreduce-client/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-openstack.jar:/usr/hdp/current/hadoop-mapreduce-client/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-mapreduce-client/log4j-1.2.17.jar:/usr/hdp/current/hadoop-mapreduce-client/jersey-core-1.9.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-common-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/commons-io-2.4.jar:/usr/hdp/current/hadoop-mapreduce-client/commons-net-3.1.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/xmlenc-0.52.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-sls-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins.jar:/usr/hdp/current/hadoop-mapreduce-client/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-common.jar:/usr/hdp/current/hadoop-mapreduce-client/paranamer-2.3.jar:/usr/hdp/current/hadoop-mapreduce-client/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-ant-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-mapreduce-client/commons-codec-1.4.jar:/usr/hdp/current/hadoop-mapreduce-client/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-app.jar:/usr/hdp/current/hadoop-mapreduce-client/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-mapreduce-client/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-gridmix-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-mapreduce-client/jetty-util-6.1.26.jar:/usr/hdp/current/hadoop-mapreduce-client/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-2.7.0.2.3.0.0-1551-tests.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-rumen-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-mapreduce-client/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-distcp-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/guava-11.0.2.jar:/usr/hdp/current/hadoop-mapreduce-client/xz-1.0.jar:/usr/hdp/current/hadoop-mapreduce-client/joda-time-2.7.jar:/usr/hdp/current/hadoop-mapreduce-client/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-mapreduce-client/commons-lang-2.6.jar:/usr/hdp/current/hadoop-mapreduce-client/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hadoop-mapreduce-client/aws-java-sdk-1.7.4.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.jar:/usr/hdp/current/hadoop-mapreduce-client/httpclient-4.2.5.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-auth.jar:/usr/hdp/current/hadoop-mapreduce-client/servlet-api-2.5.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-app-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-mapreduce-client/jsch-0.1.42.jar:/usr/hdp/current/hadoop-mapreduce-client/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-mapreduce-client/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-azure-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-mapreduce-client/asm-3.2.jar:/usr/hdp/current/hadoop-mapreduce-client/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-mapreduce-client/commons-lang3-3.3.2.jar:/usr/hdp/current/hadoop-mapreduce-client/commons-httpclient-3.1.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/jsp-api-2.1.jar:/usr/hdp/current/hadoop-mapreduce-client/metrics-core-3.0.1.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-azure.jar:/usr/hdp/current/hadoop-mapreduce-client/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-mapreduce-client/azure-storage-2.0.0.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-openstack-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.jar:/usr/hdp/current/tez-client/tez-yarn-timeline-history-with-acls-0.7.0.2.3.0.0-1551.jar:/usr/hdp/current/tez-client/tez-api-0.7.0.2.3.0.0-1551.jar:/usr/hdp/current/tez-client/tez-tests-0.7.0.2.3.0.0-1551.jar:/usr/hdp/current/tez-client/tez-yarn-timeline-history-0.7.0.2.3.0.0-1551.jar:/usr/hdp/current/tez-client/tez-examples-0.7.0.2.3.0.0-1551.jar:/usr/hdp/current/tez-client/tez-runtime-internals-0.7.0.2.3.0.0-1551.jar:/usr/hdp/current/tez-client/tez-mbeans-resource-calculator-0.7.0.2.3.0.0-1551.jar:/usr/hdp/current/tez-client/tez-dag-0.7.0.2.3.0.0-1551.jar:/usr/hdp/current/tez-client/tez-common-0.7.0.2.3.0.0-1551.jar:/usr/hdp/current/tez-client/tez-mapreduce-0.7.0.2.3.0.0-1551.jar:/usr/hdp/current/tez-client/tez-runtime-library-0.7.0.2.3.0.0-1551.jar:/usr/hdp/current/tez-client/lib/jsr305-2.0.3.jar:/usr/hdp/current/tez-client/lib/hadoop-mapreduce-client-core-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/tez-client/lib/commons-collections-3.2.1.jar:/usr/hdp/current/tez-client/lib/jersey-client-1.9.jar:/usr/hdp/current/tez-client/lib/jetty-6.1.26.jar:/usr/hdp/current/tez-client/lib/commons-cli-1.2.jar:/usr/hdp/current/tez-client/lib/slf4j-log4j12-1.7.5.jar:/usr/hdp/current/tez-client/lib/hadoop-yarn-server-web-proxy-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/tez-client/lib/jersey-json-1.9.jar:/usr/hdp/current/tez-client/lib/jettison-1.3.4.jar:/usr/hdp/current/tez-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/tez-client/lib/hadoop-mapreduce-client-common-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/tez-client/lib/commons-io-2.4.jar:/usr/hdp/current/tez-client/lib/commons-codec-1.4.jar:/usr/hdp/current/tez-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/tez-client/lib/jetty-util-6.1.26.jar:/usr/hdp/current/tez-client/lib/hadoop-annotations-2.7.0.2.3.0.0-1551.jar:/usr/hdp/current/tez-client/lib/commons-collections4-4.0.jar:/usr/hdp/current/tez-client/lib/slf4j-api-1.7.5.jar:/usr/hdp/current/tez-client/lib/guava-11.0.2.jar:/usr/hdp/current/tez-client/lib/commons-lang-2.6.jar:/usr/hdp/current/tez-client/lib/servlet-api-2.5.jar:/etc/tez/conf/:/usr/hdp/2.3.0.0-1551/tez/tez-yarn-timeline-history-with-acls-0.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/tez/tez-api-0.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/tez/tez-tests-0.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/tez/tez-yarn-timeline-history-0.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/tez/tez-examples-0.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/tez/tez-runtime-internals-0.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/tez/tez-mbeans-resource-calculator-0.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/tez/tez-dag-0.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/tez/tez-common-0.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/tez/tez-mapreduce-0.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/tez/tez-runtime-library-0.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/tez/lib/jsr305-2.0.3.jar:/usr/hdp/2.3.0.0-1551/tez/lib/hadoop-mapreduce-client-core-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/tez/lib/commons-collections-3.2.1.jar:/usr/hdp/2.3.0.0-1551/tez/lib/jersey-client-1.9.jar:/usr/hdp/2.3.0.0-1551/tez/lib/jetty-6.1.26.jar:/usr/hdp/2.3.0.0-1551/tez/lib/commons-cli-1.2.jar:/usr/hdp/2.3.0.0-1551/tez/lib/slf4j-log4j12-1.7.5.jar:/usr/hdp/2.3.0.0-1551/tez/lib/hadoop-yarn-server-web-proxy-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/tez/lib/jersey-json-1.9.jar:/usr/hdp/2.3.0.0-1551/tez/lib/jettison-1.3.4.jar:/usr/hdp/2.3.0.0-1551/tez/lib/commons-math3-3.1.1.jar:/usr/hdp/2.3.0.0-1551/tez/lib/hadoop-mapreduce-client-common-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/tez/lib/commons-io-2.4.jar:/usr/hdp/2.3.0.0-1551/tez/lib/commons-codec-1.4.jar:/usr/hdp/2.3.0.0-1551/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.3.0.0-1551/tez/lib/jetty-util-6.1.26.jar:/usr/hdp/2.3.0.0-1551/tez/lib/hadoop-annotations-2.7.0.2.3.0.0-1551.jar:/usr/hdp/2.3.0.0-1551/tez/lib/commons-collections4-4.0.jar:/usr/hdp/2.3.0.0-1551/tez/lib/slf4j-api-1.7.5.jar:/usr/hdp/2.3.0.0-1551/tez/lib/guava-11.0.2.jar:/usr/hdp/2.3.0.0-1551/tez/lib/commons-lang-2.6.jar:/usr/hdp/2.3.0.0-1551/tez/lib/servlet-api-2.5.jar:/etc/tez/conf\r\nSTARTUP_MSG:   build = git@github.com:hortonworks/hadoop.git -r 4e9829c5bf9d92aec68e22165306a1213d3b04ba; compiled by 'jenkins' on 2015-04-09T20:12Z\r\nSTARTUP_MSG:   java = 1.7.0_75\r\n************************************************************/\r\n15/04/10 02:31:30 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]\r\n15/04/10 02:31:30 INFO namenode.NameNode: createNameNode [-bootstrapStandby, -nonInteractive]\r\n15/04/10 02:31:31 WARN common.Util: Path /grid/0/hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration.\r\n15/04/10 02:31:31 WARN common.Util: Path /grid/0/hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration.\r\n15/04/10 02:31:33 INFO ipc.Client: Retrying connect to server: devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)\r\n15/04/10 02:31:34 INFO ipc.Client: Retrying connect to server: devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125:8020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)\r\n15/04/10 02:31:35 INFO ipc.Client: Retrying connect to server: devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125:8020. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)\r\n15/04/10 02:31:36 INFO ipc.Client: Retrying connect to server: devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125:8020. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)\r\n15/04/10 02:31:37 INFO ipc.Client: Retrying connect to server: devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125:8020. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)\r\n15/04/10 02:31:38 INFO ipc.Client: Retrying connect to server: devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125:8020. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)\r\n15/04/10 02:31:39 INFO ipc.Client: Retrying connect to server: devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125:8020. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)\r\n15/04/10 02:31:40 INFO ipc.Client: Retrying connect to server: devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125:8020. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)\r\n15/04/10 02:31:41 INFO ipc.Client: Retrying connect to server: devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125:8020. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)\r\n15/04/10 02:31:42 INFO ipc.Client: Retrying connect to server: devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125:8020. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)\r\n15/04/10 02:31:43 INFO ipc.Client: Retrying connect to server: devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125:8020. Already tried 10 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)\r\n15/04/10 02:31:44 INFO ipc.Client: Retrying connect to server: devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125:8020. Already tried 11 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)\r\n15/04/10 02:31:45 INFO ipc.Client: Retrying connect to server: devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125:8020. Already tried 12 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)\r\n15/04/10 02:31:46 INFO ipc.Client: Retrying connect to server: devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125:8020. Already tried 13 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)\r\n15/04/10 02:31:47 INFO ipc.Client: Retrying connect to server: devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125:8020. Already tried 14 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)\r\n=====================================================\r\nAbout to bootstrap Standby ID nn1 from:\r\n           Nameservice ID: mycluster\r\n        Other Namenode ID: nn2\r\n  Other NN's HTTP address: http://devdeploy-os-amb-r6-secha-1428629670-6.novalocal:50070\r\n  Other NN's IPC  address: devdeploy-os-amb-r6-secha-1428629670-6.novalocal/192.168.75.125:8020\r\n             Namespace ID: 356010440\r\n            Block pool ID: BP-413857717-192.168.75.125-1428633097637\r\n               Cluster ID: CID-ff355b98-2636-4b11-a7d9-aecfb9458fe1\r\n           Layout version: -63\r\n=====================================================\r\n15/04/10 02:31:49 INFO common.Storage: Storage directory /grid/0/hadoop/hdfs/namenode has been successfully formatted.\r\n15/04/10 02:31:49 WARN common.Util: Path /grid/0/hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration.\r\n15/04/10 02:31:49 WARN common.Util: Path /grid/0/hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration.\r\n15/04/10 02:31:49 WARN common.Storage: set restore failed storage to true\r\n15/04/10 02:31:52 INFO namenode.TransferFsImage: Opening connection to http://devdeploy-os-amb-r6-secha-1428629670-6.novalocal:50070/imagetransfer?getimage=1&txid=0&storageInfo=-63:356010440:0:CID-ff355b98-2636-4b11-a7d9-aecfb9458fe1\r\n15/04/10 02:31:52 INFO namenode.TransferFsImage: Image Transfer timeout configured to 60000 milliseconds\r\n15/04/10 02:31:52 INFO namenode.TransferFsImage: Transfer took 0.02s at 0.00 KB/s\r\n15/04/10 02:31:52 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size 336 bytes.\r\n15/04/10 02:31:52 INFO util.ExitUtil: Exiting with status 0\r\n15/04/10 02:31:52 INFO namenode.NameNode: SHUTDOWN_MSG: \r\n/************************************************************\r\nSHUTDOWN_MSG: Shutting down NameNode at devdeploy-os-amb-r6-secha-1428629670-2.novalocal/192.168.75.130\r\n************************************************************/\r")
2015-04-10 02:31:52,828 - Standby namenode bootstrapped successfully
2015-04-10 02:31:52,833 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'recursive': True}
2015-04-10 02:31:53,192 - Directory['/grid/0/log/hadoop/hdfs'] {'owner': 'hdfs', 'recursive': True}
2015-04-10 02:31:53,546 - File['/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'] {'action': ['delete'], 'not_if': 'ls /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid >/dev/null 2>&1 && ps -p `cat /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid` >/dev/null 2>&1'}
2015-04-10 02:31:53,829 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /etc/hadoop/conf start namenode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'not_if': 'ls /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid >/dev/null 2>&1 && ps -p `cat /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid` >/dev/null 2>&1'}

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

AMBARI-10449.patch
11/Apr/15 00:46
5 kB
Sumit Mohanty

Activity

People

Assignee:: Sumit Mohanty

Reporter:: Sumit Mohanty

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 10/Apr/15 23:54

Updated:: 20/Apr/15 22:47

Resolved:: 20/Apr/15 22:47