Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-10417

Flume fails to restart on ubuntu 12.04 after system restart because /var/run/flume is deleted

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0, 2.1.0
    • 2.1.0
    • None
    • None

    Description

      Very similar issue to AMBARI-10317, but for flume:

      2015-04-09 17:22:09,647 - Error while executing command 'restart':
      Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 214, in execute
      method(env)
      File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 362, in restart
      self.stop(env)
      File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
      return fn(*args, **kwargs)
      File "/var/lib/ambari-agent/cache/common-services/FLUME/1.4.0.2.0/package/scripts/flume_handler.py", line 70, in stop
      flume(action='stop')
      File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
      return fn(*args, **kwargs)
      File "/var/lib/ambari-agent/cache/common-services/FLUME/1.4.0.2.0/package/scripts/flume.py", line 167, in flume
      _set_desired_state('INSTALLED')
      File "/var/lib/ambari-agent/cache/common-services/FLUME/1.4.0.2.0/package/scripts/flume.py", line 244, in _set_desired_state
      content = state,
      File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in _init_
      self.env.run()
      File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
      self.run_action(resource, action)
      File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
      provider_action()
      File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 105, in action_create
      raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
      Fail: Applying u"File['/var/run/flume/ambari-state.txt']" failed, parent directory /var/run/flume doesn't exist
      stdout: /var/lib/ambari-agent/data/output-270.txt

      2015-04-09 17:22:05,187 - u"Group['hadoop']"

      {'ignore_failures': False}

      2015-04-09 17:22:05,187 - Modifying group hadoop
      2015-04-09 17:22:05,238 - u"Group['users']"

      {'ignore_failures': False}

      2015-04-09 17:22:05,238 - Modifying group users
      2015-04-09 17:22:05,283 - u"Group['knox']"

      {'ignore_failures': False}

      2015-04-09 17:22:05,283 - Modifying group knox
      2015-04-09 17:22:05,326 - u"Group['ranger']"

      {'ignore_failures': False}

      2015-04-09 17:22:05,326 - Modifying group ranger
      2015-04-09 17:22:05,369 - u"User['hive']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}

      2015-04-09 17:22:05,370 - Modifying user hive
      2015-04-09 17:22:05,412 - u"User['oozie']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}

      2015-04-09 17:22:05,413 - Modifying user oozie
      2015-04-09 17:22:05,458 - u"User['root']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}

      2015-04-09 17:22:05,458 - Modifying user root
      2015-04-09 17:22:05,502 - u"User['ambari-qa']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}

      2015-04-09 17:22:05,502 - Modifying user ambari-qa
      2015-04-09 17:22:05,545 - u"User['flume']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}

      2015-04-09 17:22:05,545 - Modifying user flume
      2015-04-09 17:22:05,588 - u"User['hdfs']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}

      2015-04-09 17:22:05,589 - Modifying user hdfs
      2015-04-09 17:22:05,633 - u"User['knox']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}

      2015-04-09 17:22:05,633 - Modifying user knox
      2015-04-09 17:22:05,676 - u"User['ranger']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}

      2015-04-09 17:22:05,677 - Modifying user ranger
      2015-04-09 17:22:05,722 - u"User['mapred']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}

      2015-04-09 17:22:05,723 - Modifying user mapred
      2015-04-09 17:22:05,766 - u"User['tez']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}

      2015-04-09 17:22:05,766 - Modifying user tez
      2015-04-09 17:22:05,810 - u"User['zookeeper']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}

      2015-04-09 17:22:05,810 - Modifying user zookeeper
      2015-04-09 17:22:05,854 - u"User['rangerlogger']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}

      2015-04-09 17:22:05,854 - Modifying user rangerlogger
      2015-04-09 17:22:05,897 - u"User['sqoop']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}

      2015-04-09 17:22:05,897 - Modifying user sqoop
      2015-04-09 17:22:05,942 - u"User['yarn']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}

      2015-04-09 17:22:05,942 - Modifying user yarn
      2015-04-09 17:22:05,986 - u"User['hcat']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}

      2015-04-09 17:22:05,986 - Modifying user hcat
      2015-04-09 17:22:06,029 - u"User['ams']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}

      2015-04-09 17:22:06,029 - Modifying user ams
      2015-04-09 17:22:06,073 - u"User['rangeradmin']"

      {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}

      2015-04-09 17:22:06,073 - Modifying user rangeradmin
      2015-04-09 17:22:06,121 - u"File['/var/lib/ambari-agent/data/tmp/changeUid.sh']"

      {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}

      2015-04-09 17:22:06,388 - u"Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa']"

      {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}

      2015-04-09 17:22:06,430 - Skipping u"Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa']" due to not_if
      2015-04-09 17:22:06,431 - u"Group['hdfs']"

      {'ignore_failures': False}

      2015-04-09 17:22:06,431 - Modifying group hdfs
      2015-04-09 17:22:06,474 - u"User['hdfs']"

      {'ignore_failures': False, 'groups': [u'hadoop', 'hadoop', 'hdfs', u'hdfs']}

      2015-04-09 17:22:06,475 - Modifying user hdfs
      2015-04-09 17:22:06,520 - u"Directory['/etc/hadoop']"

      {'mode': 0755}

      2015-04-09 17:22:06,657 - u"Directory['/etc/hadoop/conf.empty']"

      {'owner': 'root', 'group': 'hadoop', 'recursive': True}

      2015-04-09 17:22:06,798 - u"Link['/etc/hadoop/conf']"

      {'not_if': 'ls /etc/hadoop/conf', 'to': '/etc/hadoop/conf.empty'}

      2015-04-09 17:22:06,843 - Skipping u"Link['/etc/hadoop/conf']" due to not_if
      2015-04-09 17:22:06,852 - u"File['/etc/hadoop/conf/hadoop-env.sh']"

      {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}

      2015-04-09 17:22:07,089 - u"Execute['('setenforce', '0')']"

      {'sudo': True, 'only_if': 'test -f /selinux/enforce'}

      2015-04-09 17:22:07,141 - Skipping u"Execute['('setenforce', '0')']" due to only_if
      2015-04-09 17:22:07,142 - u"Directory['/var/log/hadoop']"

      {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}

      2015-04-09 17:22:07,557 - u"Directory['/var/run/hadoop']"

      {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'}

      2015-04-09 17:22:07,956 - u"Directory['/tmp/hadoop-hdfs']"

      {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'}

      2015-04-09 17:22:08,271 - u"File['/etc/hadoop/conf/commons-logging.properties']"

      {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}

      2015-04-09 17:22:08,499 - u"File['/etc/hadoop/conf/health_check']"

      {'content': Template('health_check-v2.j2'), 'owner': 'hdfs'}

      2015-04-09 17:22:08,723 - u"File['/etc/hadoop/conf/log4j.properties']"

      {'content': '...', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}

      2015-04-09 17:22:08,957 - u"File['/etc/hadoop/conf/hadoop-metrics2.properties']"

      {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}

      2015-04-09 17:22:09,183 - u"File['/etc/hadoop/conf/task-log4j.properties']"

      {'content': StaticFile('task-log4j.properties'), 'mode': 0755}

      2015-04-09 17:22:09,552 - u"File['/var/run/flume/ambari-state.txt']"

      {'content': 'INSTALLED'}

      2015-04-09 17:22:09,647 - Error while executing command 'restart':
      Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 214, in execute
      method(env)
      File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 362, in restart
      self.stop(env)
      File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
      return fn(*args, **kwargs)
      File "/var/lib/ambari-agent/cache/common-services/FLUME/1.4.0.2.0/package/scripts/flume_handler.py", line 70, in stop
      flume(action='stop')
      File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
      return fn(*args, **kwargs)
      File "/var/lib/ambari-agent/cache/common-services/FLUME/1.4.0.2.0/package/scripts/flume.py", line 167, in flume
      _set_desired_state('INSTALLED')
      File "/var/lib/ambari-agent/cache/common-services/FLUME/1.4.0.2.0/package/scripts/flume.py", line 244, in _set_desired_state
      content = state,
      File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in _init_
      self.env.run()
      File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
      self.run_action(resource, action)
      File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
      provider_action()
      File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 105, in action_create
      raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
      Fail: Applying u"File['/var/run/flume/ambari-state.txt']" failed, parent directory /var/run/flume doesn't exist
      2015-04-09 17:22:09,691 - Command: /usr/bin/hdp-select status flume-server > /tmp/tmpKcN1a_
      Output: flume-server - 2.2.0.0-2041

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. AMBARI-10317 Knox gateway fails to restart on Ubuntu 12.04 after system restart because /var/run/knox is deleted

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. AMBARI-10413 Knox gateway fails to restart on Ubuntu 12.04 after system restart using custom pid dir because /usr/hdp/current/knox-server/pids does not point to custom pid dir

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link ReviewBoard

          Activity

            People

              aonishuk Andrew Onischuk
              dmcwhorter David McWhorter
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: