Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-10992

Ambari is too slow to start/restart components

    XMLWordPrintableJSON

Details

    • Epic
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.0.0
    • 2.1.0
    • None
    • None
    • Ambari is too slow to start/restart

    Description

      PROBLEM REASONING
      This log of HiveServer2 restart (which took 5 minutes) brightly describes what takes the most time usually for no big reason:

      014-12-10 17:09:29,060 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:09:29,073 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:09:46,301 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz': No such file or directory
2014-12-10 17:09:46,301 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:09:46,303 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chmod  555 hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chown  hdfs hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:10:29,989 - CopyFromLocal['/usr/hdp/current/hive-client/hive.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/hive', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:10:30,017 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:10:48,275 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:07,134 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:07,135 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:16,533 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:16,534 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:29,515 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:29,516 - Execute['hadoop --con014-12-10 17:09:29,060 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:09:29,073 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:09:46,301 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz': No such file or directory
2014-12-10 17:09:46,301 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:09:46,303 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chmod  555 hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chown  hdfs hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:10:29,989 - CopyFromLocal['/usr/hdp/current/hive-client/hive.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/hive', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:10:30,017 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:10:48,275 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:07,134 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:07,135 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:16,533 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:16,534 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:29,515 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:29,516 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:45,791 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz': No such file or directory
2014-12-10 17:11:45,791 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:11:45,794 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chmod  555 hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chown  hdfs hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:12:31,703 - CopyFromLocal['/usr/hdp/current/pig-client/pig.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/pig', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:12:31,703 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:12:49,508 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:09,506 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:09,507 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:18,968 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:18,969 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:32,936 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:32,937 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:52,891 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar': No such file or directory
2014-12-10 17:13:52,892 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:13:52,904 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod  555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown  hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:03,832 - Skipping Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod  555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown  hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] due to not_if
2014-12-10 17:14:03,833 - CopyFromLocal['/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/mapreduce', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:14:03,836 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:12,682 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:22,350 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:22,352 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:34,163 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:34,164 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:50,851 - Could not find file: /usr/hdp/current/sqoop-client/sqoop.tar.gz
2014-12-10 17:14:50,862 - XmlConfig['webhcat-site.xml'] {'owner': 'hcat', 'group': 'hadoop', 'conf_dir': '/etc/hive-webhcat/conf', 'configuration_attributes': ..., 'configurations': ...}
2014-12-10 17:14:50,979 - Generating config: /etc/hive-webhcat/conf/webhcat-site.xml
2014-12-10 17:14:50,980 - File['/etc/hive-webhcat/conf/webhcat-site.xml'] {'owner': 'hcat', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2014-12-10 17:14:50,983 - Writing File['/etc/hive-webhcat/conf/webhcat-site.xml'] because it doesn't exist
2014-12-10 17:14:51,114 - Changing owner for /etc/hive-webhcat/conf/webhcat-site.xml from 0 to hcat
2014-12-10 17:14:51,169 - Changing group for /etc/hive-webhcat/conf/webhcat-site.xml from 0 to hadoop
2014-12-10 17:14:51,221 - File['/etc/hive-webhcat/conf/webhcat-env.sh'] {'content': InlineTemplate(...), 'owner': 'hcat', 'group': 'hadoop'}
2014-12-10 17:14:51,222 - Writing File['/etc/hive-webhcat/conf/webhcat-env.sh'] because it doesn't exist
2014-12-10 17:14:51,312 - Changing owner for /etc/hive-webhcat/conf/webhcat-env.sh from 0 to hcat
2014-12-10 17:14:51,367 - Changing group for /etc/hive-webhcat/conf/webhcat-env.sh from 0 to hadoop
2014-12-10 17:14:51,423 - Execute['env HADOOP_HOME=/usr/hdp/current/hadoop-client /usr/hdp/current/hive-webhcat/sbin/webhcat_server.sh start'] {'not_if': 'ls /var/run/webhcat/webhcat.pid >/dev/null 2>&1 && ps -p `cat /var/run/webhcat/webhcat.pid` >/dev/null 2>&1', 'user': 'hcat'}fig /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:45,791 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz': No such file or directory
2014-12-10 17:11:45,791 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:11:45,794 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chmod  555 hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chown  hdfs hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:12:31,703 - CopyFromLocal['/usr/hdp/current/pig-client/pig.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/pig', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:12:31,703 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:12:49,508 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:09,506 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:09,507 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:18,968 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:18,969 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:32,936 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:32,937 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:52,891 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar': No such file or directory
2014-12-10 17:13:52,892 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:13:52,904 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod  555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown  hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:03,832 - Skipping Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod  555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown  hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] due to not_if
2014-12-10 17:14:03,833 - CopyFromLocal['/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/mapreduce', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:14:03,836 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:12,682 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:22,350 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:22,352 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:34,163 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:34,164 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:50,851 - Could not find file: /usr/hdp/current/sqoop-client/sqoop.tar.gz

      The problem is ‘hadoop fs’ calls. One call takes 3-30 seconds depending on node load CPU/RAM etc. Mostly this is because every time call is done new jvm is created and probably due to some DN/NN communication is redone. Since we have more and more this calls triggered with every release, this became a problem now.

      BASIC SOLUTION DESCRIPTION
      Instead of copying files to hdfs and creating directories one by one, by calling countless number of ‘hadoop fs’ calls, we can do one single execution of the jar file which will do all the creating it is asked.

      For example:

      hadoop -fs -mkdir /a
hadoop -fs -chmod 755 /a
hadoop -fs -chown ambari-qa /b
hadoop -fs -copyFromLocal /x /y

      Can be replaced with
      resources.json like this which is passed to our new jar (below):

      {
  {
    “path”:”/a”
    “type”: directory”
    “mode”: “755”
  },
  {
    “path”:”/b”
    “type”: directory”
    “owner”: “ambari-qa”
  },
  {
    “path”:”/y”
    “type”: directory”
    “source”: “/x”
  },
}

      which will handle all that only in one single time-expensive call:
      hadoop jar create-hdfs-resources.jar resources.json

      For synchronization between nodes (to know who and when executes this time expensive calls) ZK is used for more details please read the doc below

      Attachments

        Issue Links

          links to

          Web Link Implementation doc

          Activity

            People

              aonishuk Andrew Onischuk
              aonishuk Andrew Onischuk
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: