Uploaded image for project: 'Metron (Retired)'
  1. Metron (Retired)
  2. METRON-1410

Some more upgrade fallout... Can't restart Metron Indexing.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Done
    • Major
    • Resolution: Done
    • None
    • 0.5.0
    • None

    Description

      After upgrading from 0.4.1 to 0.4.2, I can't seem to start or restart Metron Indexing. I get the following errors:

      stderr:   /var/lib/ambari-agent/data/errors-2468.txt

      Traceback (most recent call last):
        File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/indexing_master.py", line 160, in <module>
          Indexing().execute()
        File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
          method(env)
        File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/indexing_master.py", line 82, in start
          self.configure(env)
        File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/indexing_master.py", line 72, in configure
          commands.create_hbase_tables()
        File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/indexing_commands.py", line 126, in create_hbase_tables
          user=self.__params.hbase_user
        File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in _init_
          self.env.run()
        File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
          self.run_action(resource, action)
        File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
          provider_action()
        File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run
          tries=self.resource.tries, try_sleep=self.resource.try_sleep)
        File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
          result = function(command, **kwargs)
        File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
          tries=tries, try_sleep=try_sleep)
        File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
          result = _call(command, **kwargs_copy)
        File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 293, in _call
          raise ExecutionFailed(err_msg, code, out, err)
      resource_management.core.exceptions.ExecutionFailed: Execution of 'echo "create 'metron_update','t'" | hbase shell -n' returned 1. ERROR RuntimeError: Table already exists: metron_update!

      stdout:   /var/lib/ambari-agent/data/output-2468.txt

      2018-01-18 16:54:30,101 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
      2018-01-18 16:54:30,301 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
      2018-01-18 16:54:30,302 - Group['metron'] {}
      2018-01-18 16:54:30,303 - Group['livy'] {}
      2018-01-18 16:54:30,303 - Group['elasticsearch'] {}
      2018-01-18 16:54:30,303 - Group['spark'] {}
      2018-01-18 16:54:30,303 - Group['zeppelin'] {}
      2018-01-18 16:54:30,304 - Group['hadoop'] {}
      2018-01-18 16:54:30,304 - Group['kibana'] {}
      2018-01-18 16:54:30,304 - Group['users'] {}
      2018-01-18 16:54:30,304 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,305 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,306 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,306 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,307 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,307 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
      2018-01-18 16:54:30,308 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,309 - User['metron'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,309 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,310 - User['elasticsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,310 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,311 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
      2018-01-18 16:54:30,311 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,312 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,312 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,313 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,314 - User['kibana'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,314 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,315 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,315 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
      2018-01-18 16:54:30,316 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
      2018-01-18 16:54:30,317 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
      2018-01-18 16:54:30,323 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
      2018-01-18 16:54:30,324 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
      2018-01-18 16:54:30,325 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
      2018-01-18 16:54:30,326 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
      2018-01-18 16:54:30,331 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
      2018-01-18 16:54:30,332 - Group['hdfs'] {}
      2018-01-18 16:54:30,332 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']}
      2018-01-18 16:54:30,333 - FS Type:
      2018-01-18 16:54:30,333 - Directory['/etc/hadoop'] {'mode': 0755}
      2018-01-18 16:54:30,346 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
      2018-01-18 16:54:30,347 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
      2018-01-18 16:54:30,365 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
      2018-01-18 16:54:30,384 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
      2018-01-18 16:54:30,386 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
      2018-01-18 16:54:30,386 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
      2018-01-18 16:54:30,390 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
      2018-01-18 16:54:30,392 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
      2018-01-18 16:54:30,393 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
      2018-01-18 16:54:30,403 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs', 'group': 'hadoop'}
      2018-01-18 16:54:30,404 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
      2018-01-18 16:54:30,405 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
      2018-01-18 16:54:30,409 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
      2018-01-18 16:54:30,414 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
      2018-01-18 16:54:30,667 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
      2018-01-18 16:54:30,669 - Running indexing configure
      2018-01-18 16:54:30,676 - File['/usr/metron/0.4.2/config/elasticsearch.properties'] {'owner': 'metron', 'content': Template('elasticsearch.properties.j2'), 'group': 'metron'}
      2018-01-18 16:54:30,678 - Patch global config in Zookeeper
      2018-01-18 16:54:30,678 - Setup temporary global config JSON patch (formatting per RFC6902): /tmp/metron-global-config-patch.json
      2018-01-18 16:54:30,681 - File['/tmp/metron-global-config-patch.json'] {'owner': 'metron', 'content': InlineTemplate(...), 'group': 'metron'}
      2018-01-18 16:54:30,681 - Patching global config in ZooKeeper
      2018-01-18 16:54:30,681 - Execute['/usr/metron/0.4.2/bin/zk_load_configs.sh --zk_quorum metron1:2181,metron2:2181 --mode PATCH --config_type GLOBAL --patch_file /tmp/metron-global-config-patch.json'] {'path': ['/usr/jdk64/jdk1.8.0_77/bin']}
      2018-01-18 16:55:19,874 - Done patching global config
      2018-01-18 16:55:19,874 - Pull zookeeper config locally
      2018-01-18 16:55:19,874 - Pulling all Metron configs down from ZooKeeper to local file system
      2018-01-18 16:55:19,874 - NOTE - THIS IS OVERWRITING THE LOCAL METRON CONFIG DIR WITH ZOOKEEPER CONTENTS: /usr/metron/0.4.2/config/zookeeper
      2018-01-18 16:55:19,875 - Execute['/usr/metron/0.4.2/bin/zk_load_configs.sh --zk_quorum metron1:2181,metron2:2181 --mode PULL --output_dir /usr/metron/0.4.2/config/zookeeper --force'] {'path': ['/usr/jdk64/jdk1.8.0_77/bin']}
      2018-01-18 16:56:08,286 - Creating HBase Tables for indexing
      2018-01-18 16:56:08,287 - Execute['echo "create 'metron_update','t'" | hbase shell -n'] {'logoutput': False, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3, 'user': 'hbase', 'try_sleep': 5}
      2018-01-18 16:56:59,943 - Retrying after 5 seconds. Reason: Execution of 'echo "create 'metron_update','t'" | hbase shell -n' returned 1. ERROR RuntimeError: Table already exists: metron_update!
      2018-01-18 16:57:57,884 - Retrying after 5 seconds. Reason: Execution of 'echo "create 'metron_update','t'" | hbase shell -n' returned 1. ERROR RuntimeError: Table already exists: metron_update!

      Command failed after 1 tries

      Attachments

        Issue Links

          links to

          Web Link GitHub Pull Request #901

          Activity

            People

              otto Otto Fowler
              laurensv Laurens Vets
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: