GitHub user purechoc opened a pull request:

https://github.com/apache/zeppelin/pull/1587

using pyspark(python3) with livy interperter

1. 1. 1. What is this PR for?
         using pyspark(python3) with livy interperter

1. 1. 1. What type of PR is it?
         Improvement

1. 1. 1. Todos

1. 1. 1. What is the Jira issue?
         https://issues.apache.org/jira/browse/ZEPPELIN-1609

1. 1. 1. How should this be tested?
         %livy.pyspark3
         3/2
         check result is "1.5"

1. 1. 1. Screenshots (if appropriate)

1. 1. 1. Questions:

• Does the licenses files need update? - NO
• Is there breaking changes for older versions? - NO
• Does this needs documentation? - NO

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/purechoc/zeppelin zeppelin-livy-pyspark3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1587.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1587

commit c0babf619006cc3a4ac66ffb0904b49e37f3a2ae
Author: purechoc <purechoc.en@gmail.com>
Date: 2016-11-03T07:29:43Z

add %livy.python3

Github user felixcheung commented on the issue:

https://github.com/apache/zeppelin/pull/1587

Hmm, can't we do that with the existing `%livy.pyspark` interpreter?

Github user purechoc commented on the issue:

https://github.com/apache/zeppelin/pull/1587

can't do that,
%livy.pyspark support python2 only.

Github user Leemoonsoo commented on the issue:

https://github.com/apache/zeppelin/pull/1587

@purechoc Thanks for the contribution.

Like @felixcheung mentioned, alternative approach could be adding a interpreter property [here](https://github.com/apache/zeppelin/blob/master/livy/src/main/resources/interpreter-setting.json#L145 ) and let LivyPySparkInterpreter be configured with the property without adding new interpreter class.

What do you think?

Github user purechoc commented on the issue:

https://github.com/apache/zeppelin/pull/1587

@Leemoonsoo
thanks to review.

first. i try like that.
```

1. add properties
   {
   "group": "livy",
   "name": "pyspark3",
   "className": "org.apache.zeppelin.livy.LivyPySparkInterpreter",
   "properties":
   Unknown macro: { "zeppelin.livy.kind"}

   ,
   "editor":

   { "language": "python", "editOnDblClick": false }

   },
   ```
   it is work. but that property assign both "livy.pyspark" and "livy.pyspark3".

if user want to python2 and python3 both,
user need to make new interpreter base on livy group and set property zeppelin.livy.kind=pyspark3.

if, this is better than adding new interpreter class, i fixed code and commit soon.

Github user Leemoonsoo commented on the issue:

https://github.com/apache/zeppelin/pull/1587

Thanks for explain. but i'm little bit confused.
If you are creating `LivyPySpark3Interpreter` in addition to `LivyPySparkInterpreter`, there will be no need to add property `zeppelin.livy.kind`.

But by adding a property, `LivyPySparkInterpreter` interpreter can read property and change it's behavior which version of python to use, in runtime. and no need to create `LivyPySpark3Interpreter`.
In this case, to use both python2 and python3 at the same time, user can create multiple interpreter settings in the GUI and use them.

I have no strong opinion on any approaches, but at least `python`, `pyspark` interpreter handle python version using property instead of having separate interpreter class.

@purechoc @zjffdu @prabhjyotsingh @felixcheung What do you think?

Github user purechoc commented on the issue:

https://github.com/apache/zeppelin/pull/1587

@Leemoonsoo .
thanks to comment.

you mean like this?

if user add property `zeppelin.livy.kind` = pyspark, "%livy.pyspark" working on `python2`.
if user add property `zeppelin.livy.kind` = pyspark3, "%livy.pyspark" working on `python3`.

like a set `zeppelin.python` = python or python3 ``pyspark `interpreter`
but if user want to `python2 `and `python3 `same time `python` interpreter. user need to create new interperter in the GUI.
```
interperter name = python3
interperter group = python
zeppelin.python = python3

%python3
2/3 = 1.5
```

if live using this approch, user using livy interperter like this.
```

1. run python2
   %livy.pyspark

1. run python3
   %livy_python3.pyspark
   ```

add property `zeppelin.livy.kind` and set python version if user want to run. (each time. not same time)
is this right?

Github user Leemoonsoo commented on the issue:

https://github.com/apache/zeppelin/pull/1587

I see what you mean. Thanks for the explanation. Make sense.
LGTM

Github user felixcheung commented on the issue:

https://github.com/apache/zeppelin/pull/1587

I think I can review this in more details, but is there a need to run both Python2 and Python 3 at the same time in the same environment? Most likely people will have chosen one?

If one at a time is ok why not do this with an environment variable LIVY_PYTHON - this is the similar approach Spark has chosen - and must also apply here since Livy just kick off PySpark anyway. So there really only one can be set at a time.

Or am I missing something?

Github user purechoc commented on the issue:

https://github.com/apache/zeppelin/pull/1587

for example.
in our environment. (zeppelin + livy)
there is 3 team, (teamA, B, C)
teamA using python2,
teamB, C using python3,
there are using same enveroment (zepplein + livy)

so. livy offer pyspark and pyspark3. this is livy's feature.
watch this : https://github.com/cloudera/livy/commit/d3d203160436abb37b1136035d534f3d1c1a0533

i dont't understand livy.pyspark and livy.pyspar3 using same time.
(pyspark3 is livy's kind feature)

Github user felixcheung commented on the issue:

https://github.com/apache/zeppelin/pull/1587

Ah thanks for pointing out. I will look into what Livy is doing shortly

Github user purechoc commented on the issue:

https://github.com/apache/zeppelin/pull/1587

@zjffdu @felixcheung @Leemoonsoo
could you merge about this?
or need to fix something?

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1587

I prefer to use `%livy.pyspark` & `$livy.pyspark3` instead of `zeppelin.livy.kind`. Because if user change `zeppelin.livy.kind` some paragraphs may not work again, this is very confusing. So explicitly specifying python version is better IMO. And could you wait for #1612 which is refactoring livy interpreter ? After #1612 , it should be pretty easy to add `%livy.pyspark3`

Github user purechoc commented on the issue:

https://github.com/apache/zeppelin/pull/1587

@zjffdu
thanks to comment.
i am wait #1612
#1612 is LGTM

Github user purechoc commented on the issue:

https://github.com/apache/zeppelin/pull/1587

refector code based on #1612

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1587

lgtm

Github user purechoc commented on the issue:

https://github.com/apache/zeppelin/pull/1587

@felixcheung @Leemoonsoo
please review this.
thanks

Github user felixcheung commented on the issue:

https://github.com/apache/zeppelin/pull/1587

shouldn't this be listed here: https://github.com/apache/zeppelin/blob/master/conf/zeppelin-site.xml.template#L193

Github user purechoc commented on the issue:

https://github.com/apache/zeppelin/pull/1587

thank to address.

update code completed

Github user felixcheung commented on the issue:

https://github.com/apache/zeppelin/pull/1587

LGTM
merging if no more comment

Github user purechoc closed the pull request at:

https://github.com/apache/zeppelin/pull/1587

GitHub user purechoc reopened a pull request:

https://github.com/apache/zeppelin/pull/1587

ZEPPELIN-1609 using pyspark(python3) with livy interperter

1. 1. 1. What is this PR for?
         using pyspark(python3) with livy interperter

1. 1. 1. What type of PR is it?
         Improvement

1. 1. 1. Todos

1. 1. 1. What is the Jira issue?
         https://issues.apache.org/jira/browse/ZEPPELIN-1609

1. 1. 1. How should this be tested?
         %livy.pyspark3
         3/2
         check result is "1.5"

1. 1. 1. Screenshots (if appropriate)

1. 1. 1. Questions:

• Does the licenses files need update? - NO
• Is there breaking changes for older versions? - NO
• Does this needs documentation? - NO

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/purechoc/zeppelin zeppelin-livy-pyspark3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1587.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1587

commit c0babf619006cc3a4ac66ffb0904b49e37f3a2ae
Author: purechoc <purechoc.en@gmail.com>
Date: 2016-11-03T07:29:43Z

add %livy.python3

commit 3775e47158c7146a8d005af851dd17ca6bfaca7b
Author: purechoc <purechoc.en@gmail.com>
Date: 2016-11-03T10:25:07Z

add %livy.python3

commit 7e4210807bcf7483240b06f11081d7cb07e79b88
Author: purechoc <purechoc.en@gmail.com>
Date: 2016-11-03T10:49:25Z

add %livy.python3

commit d0e2293327f73fa8ebe89498652cc94db45a02bc
Author: purechoc <purechoc.en@gmail.com>
Date: 2016-11-03T23:56:41Z

Trigger

commit 3fc98cd47f88e59f6ae5a0c9e35d476b74df15bf
Author: purechoc <purechoc.en@gmail.com>
Date: 2016-11-04T05:33:45Z

add configuration

commit 51140f2686af2cf381ea25dd451097b850f2e396
Author: purechoc <purechoc.en@gmail.com>
Date: 2016-11-06T23:39:13Z

Trigger

commit e5db6405e3e87f6425a9503e4fee66167e394b7a
Author: purechoc <purechoc.en@gmail.com>
Date: 2016-11-07T00:10:29Z

add properties

commit 55d486ff5987e28f112e81d665c46c4276c47739
Author: purechoc <purechoc.en@gmail.com>
Date: 2016-11-09T00:42:24Z

Trigger

commit 476ae2504e24544f44b8bff9a8ed428a6d1b7c04
Author: purechoc <purechoc@ncsoft.com>
Date: 2016-12-11T23:38:07Z

Merge remote-tracking branch 'upstream3/master' into zeppelin-livy-pyspark3

Conflicts:
livy/src/main/java/org/apache/zeppelin/livy/LivyPySparkInterpreter.java

commit 7c0f70d6f48852b733f61272a787bd4438733ec7
Author: purechoc <purechoc@ncsoft.com>
Date: 2016-12-11T23:53:24Z

refactor code

commit 6f7391fb706efc7f7db4751ac6035ad6e634a953
Author: purechoc <purechoc@ncsoft.com>
Date: 2016-12-29T00:08:18Z

update template

Github user purechoc commented on the issue:

https://github.com/apache/zeppelin/pull/1587

i think CI failure cause is not this PR.
some other reason in other testcase.

Github user felixcheung commented on the issue:

https://github.com/apache/zeppelin/pull/1587

all green!
merging if no more comment

Github user asfgit closed the pull request at:

https://github.com/apache/zeppelin/pull/1587