[AMBARI-24587] HDI Livy2 fails to restart

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.7.3
• Labels: pull-request-available
• Assignee: Andrew Onischuk
• Reporter: Andrew Onischuk
• Created: 03/Sep/18 21:49
• Updated: 22/Oct/18 21:00
• Resolved: 16/Oct/18 11:07

Description

Livy2 restart fails from Ambari due to Ambari could not fetch some Hadoop
configs?

StdErr:

    
    Traceback (most recent call last):
      File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/livy2_server.py", line 148, in <module>
        LivyServer().execute()
      File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 351, in execute
        method(env)
      File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/livy2_server.py", line 62, in start
        self.configure(env)
      File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/livy2_server.py", line 52, in configure
        setup_livy(env, 'server', upgrade_type=upgrade_type, action = 'config')
      File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/setup_livy2.py", line 53, in setup_livy
        params.HdfsResource(None, action="execute")
      File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
        self.env.run()
      File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
        self.run_action(resource, action)
      File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
        provider_action()
      File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 681, in action_execute
        self.get_hdfs_resource_executor().action_execute(self)
      File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 164, in action_execute
        logoutput=logoutput,
      File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
        self.env.run()
      File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
        self.run_action(resource, action)
      File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
        provider_action()
      File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
        returns=self.resource.returns)
      File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
        result = function(command, **kwargs)
      File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
        tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
      File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
        result = _call(command, **kwargs_copy)
      File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
        raise ExecutionFailed(err_msg, code, out, err)
    resource_management.core.exceptions.ExecutionFailed: Execution of 'hadoop --config /usr/hdp/3.0.1.0-175/hadoop/conf jar /var/lib/ambari-agent/lib/fast-hdfs-resource.jar /var/lib/ambari-agent/tmp/hdfs_resources_1535895647.58.json' returned 1. Initializing filesystem uri: hdfs://mycluster
    Creating: Resource [source=null, target=wasb://[e-mail]/user/livy/recovery-69e01f25-3b7b-4af4-a787-37664ab45f0c, type=directory, action=create, owner=livy, group=null, mode=700, recursiveChown=false, recursiveChmod=false, changePermissionforParents=false, manageIfExists=true] in hdfs://mycluster
    Exception occurred, Reason: Wrong FS: wasb://[e-mail]/user/livy/recovery-69e01f25-3b7b-4af4-a787-37664ab45f0c, expected: hdfs://mycluster
    java.lang.IllegalArgumentException: Wrong FS: wasb://[e-mail]/user/livy/recovery-69e01f25-3b7b-4af4-a787-37664ab45f0c, expected: hdfs://mycluster
    	at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
    	at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:240)
    	at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1583)
    	at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580)
    	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1595)
    	at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1768)
    	at org.apache.ambari.fast_hdfs_resource.Resource.checkResourceParameters(Resource.java:193)
    	at org.apache.ambari.fast_hdfs_resource.Runner.main(Runner.java:112)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:498)
    	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
    	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
    

Please note that the system test sets the

property | from | to
--|-|--
`livy.server.recovery.state-store` | `zookeeper` | `filesystem`
`livy.server.recovery.state-store.url` |
`zk1-b24996.zu2zfpuge4su5ceknrmkpsq3ra.ix.internal.cloudapp.net:2181,zk2-b24996.zu2zfpuge4su5ceknrmkpsq3ra.ix.internal.cloudapp.net:2181,zk5-b24996.zu2zfpuge4su5ceknrmkpsq3ra.ix.internal.cloudapp.net:2181`

`wasb://spark2l-at30wu- [e-mail]/user/livy/recovery- 69e01f25-3b7b-4af4-a787-37664ab45f0c`

and then the restart fails.

Issue Links

• links to: GitHub Pull Request #2236
• links to: GitHub Pull Request #2237