Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-10059

Start services are failed(SERVICE_CHECK falcon is failed) when adding falcon service on kerberized Ambari 2.0 with enabled NN, HA RM

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      CentOS 6.5, hash 86f309f4bffd430f1aadab59c2f44efd27378d47

      STR:

      • Deploy cluster (HDFS, YARN/MR2, HIVE, TEZ, PIG, METRICS in my case)
      • Enable HDFS HA, RM HA
      • Enable security
      • Add Falcon Service

      Expected: Services will start successfully
      Real: Services start failed.

      stderr: /var/lib/ambari-agent/data/errors-427.txt

      2015-03-11 14:36:31,608 - Error while executing command 'service_check':
      Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 214, in execute
      method(env)
      File "/var/lib/ambari-agent/cache/common-services/FALCON/0.5.0.2.1/package/scripts/service_check.py", line 36, in service_check
      try_sleep = 20
      File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in _init_
      self.env.run()
      File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
      self.run_action(resource, action)
      File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
      provider_action()
      File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 274, in action_run
      raise ex
      Fail: Execution of '/usr/hdp/current/falcon-client/bin/falcon admin -version' returned 255. Stacktrace:
      org.apache.falcon.client.FalconCLIException: Unable to initialize Falcon Client object
      at org.apache.falcon.client.FalconClient.<init>(FalconClient.java:145)
      at org.apache.falcon.cli.FalconCLI.run(FalconCLI.java:178)
      at org.apache.falcon.cli.FalconCLI.main(FalconCLI.java:133)
      Caused by: org.apache.falcon.client.FalconCLIException: Could not authenticate, Authentication failed, status: 503, message: Service Unavailable
      at org.apache.falcon.client.FalconClient.getToken(FalconClient.java:172)
      at org.apache.falcon.client.FalconClient.<init>(FalconClient.java:143)
      ... 2 more
      Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 503, message: Service Unavailable
      at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:275)
      at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
      at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
      at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
      at org.apache.falcon.client.FalconClient.getToken(FalconClient.java:170)
      ... 3 more
      stdout: /var/lib/ambari-agent/data/output-427.txt

      2015-03-11 14:35:47,350 - u"Execute['/usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab ambari-qa']"

      {'user': 'ambari-qa'}

      2015-03-11 14:35:47,460 - u"Execute['/usr/hdp/current/falcon-client/bin/falcon admin -version']"

      {'logoutput': True, 'tries': 3, 'user': 'ambari-qa', 'try_sleep': 20}

      Stacktrace:
      org.apache.falcon.client.FalconCLIException: Unable to initialize Falcon Client object
      at org.apache.falcon.client.FalconClient.<init>(FalconClient.java:145)
      at org.apache.falcon.cli.FalconCLI.run(FalconCLI.java:178)
      at org.apache.falcon.cli.FalconCLI.main(FalconCLI.java:133)
      Caused by: org.apache.falcon.client.FalconCLIException: Could not authenticate, Authentication failed, status: 503, message: Service Unavailable
      at org.apache.falcon.client.FalconClient.getToken(FalconClient.java:172)
      at org.apache.falcon.client.FalconClient.<init>(FalconClient.java:143)
      ... 2 more
      Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 503, message: Service Unavailable
      at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:275)
      at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
      at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
      at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
      at org.apache.falcon.client.FalconClient.getToken(FalconClient.java:170)
      ... 3 more
      2015-03-11 14:35:48,816 - Retrying after 20 seconds. Reason: Execution of '/usr/hdp/current/falcon-client/bin/falcon admin -version' returned 255. Stacktrace:
      org.apache.falcon.client.FalconCLIException: Unable to initialize Falcon Client object
      at org.apache.falcon.client.FalconClient.<init>(FalconClient.java:145)
      at org.apache.falcon.cli.FalconCLI.run(FalconCLI.java:178)
      at org.apache.falcon.cli.FalconCLI.main(FalconCLI.java:133)
      Caused by: org.apache.falcon.client.FalconCLIException: Could not authenticate, Authentication failed, status: 503, message: Service Unavailable
      at org.apache.falcon.client.FalconClient.getToken(FalconClient.java:172)
      at org.apache.falcon.client.FalconClient.<init>(FalconClient.java:143)
      ... 2 more
      Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 503, message: Service Unavailable
      at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:275)
      at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
      at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
      at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
      at org.apache.falcon.client.FalconClient.getToken(FalconClient.java:170)
      ... 3 more
      Stacktrace:
      org.apache.falcon.client.FalconCLIException: Unable to initialize Falcon Client object
      at org.apache.falcon.client.FalconClient.<init>(FalconClient.java:145)
      at org.apache.falcon.cli.FalconCLI.run(FalconCLI.java:178)
      at org.apache.falcon.cli.FalconCLI.main(FalconCLI.java:133)
      Caused by: org.apache.falcon.client.FalconCLIException: Could not authenticate, Authentication failed, status: 503, message: Service Unavailable
      at org.apache.falcon.client.FalconClient.getToken(FalconClient.java:172)
      at org.apache.falcon.client.FalconClient.<init>(FalconClient.java:143)
      ... 2 more
      Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 503, message: Service Unavailable
      at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:275)
      at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
      at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
      at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
      at org.apache.falcon.client.FalconClient.getToken(FalconClient.java:170)
      ... 3 more
      2015-03-11 14:36:10,190 - Retrying after 20 seconds. Reason: Execution of '/usr/hdp/current/falcon-client/bin/falcon admin -version' returned 255. Stacktrace:
      org.apache.falcon.client.FalconCLIException: Unable to initialize Falcon Client object
      at org.apache.falcon.client.FalconClient.<init>(FalconClient.java:145)
      at org.apache.falcon.cli.FalconCLI.run(FalconCLI.java:178)
      at org.apache.falcon.cli.FalconCLI.main(FalconCLI.java:133)
      Caused by: org.apache.falcon.client.FalconCLIException: Could not authenticate, Authentication failed, status: 503, message: Service Unavailable
      at org.apache.falcon.client.FalconClient.getToken(FalconClient.java:172)
      at org.apache.falcon.client.FalconClient.<init>(FalconClient.java:143)
      ... 2 more
      Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 503, message: Service Unavailable
      at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:275)
      at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
      at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
      at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
      at org.apache.falcon.client.FalconClient.getToken(FalconClient.java:170)
      ... 3 more
      Stacktrace:
      org.apache.falcon.client.FalconCLIException: Unable to initialize Falcon Client object
      at org.apache.falcon.client.FalconClient.<init>(FalconClient.java:145)
      at org.apache.falcon.cli.FalconCLI.run(FalconCLI.java:178)
      at org.apache.falcon.cli.FalconCLI.main(FalconCLI.java:133)
      Caused by: org.apache.falcon.client.FalconCLIException: Could not authenticate, Authentication failed, status: 503, message: Service Unavailable
      at org.apache.falcon.client.FalconClient.getToken(FalconClient.java:172)
      at org.apache.falcon.client.FalconClient.<init>(FalconClient.java:143)
      ... 2 more
      Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 503, message: Service Unavailable
      at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:275)
      at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
      at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
      at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
      at org.apache.falcon.client.FalconClient.getToken(FalconClient.java:170)
      ... 3 more
      2015-03-11 14:36:31,608 - Error while executing command 'service_check':
      Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 214, in execute
      method(env)
      File "/var/lib/ambari-agent/cache/common-services/FALCON/0.5.0.2.1/package/scripts/service_check.py", line 36, in service_check
      try_sleep = 20
      File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in _init_
      self.env.run()
      File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
      self.run_action(resource, action)
      File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
      provider_action()
      File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 274, in action_run
      raise ex
      Fail: Execution of '/usr/hdp/current/falcon-client/bin/falcon admin -version' returned 255. Stacktrace:
      org.apache.falcon.client.FalconCLIException: Unable to initialize Falcon Client object
      at org.apache.falcon.client.FalconClient.<init>(FalconClient.java:145)
      at org.apache.falcon.cli.FalconCLI.run(FalconCLI.java:178)
      at org.apache.falcon.cli.FalconCLI.main(FalconCLI.java:133)
      Caused by: org.apache.falcon.client.FalconCLIException: Could not authenticate, Authentication failed, status: 503, message: Service Unavailable
      at org.apache.falcon.client.FalconClient.getToken(FalconClient.java:172)
      at org.apache.falcon.client.FalconClient.<init>(FalconClient.java:143)
      ... 2 more
      Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 503, message: Service Unavailable
      at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:275)
      at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
      at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
      at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
      at org.apache.falcon.client.FalconClient.getToken(FalconClient.java:170)
      ... 3 more

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                aonishuk Andrew Onischuk
                Reporter:
                aonishuk Andrew Onischuk
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: