Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-1655

Dynamic forms in Python interpreter do not work

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.7.0
    • Component/s: python-interpreter
    • Labels:
      None

      Description

      ZEPPELIN-1345 updates Python `PyZeppelinContext` and Pyspark `PyZeppelinContext ` but does not touch Python `Py4jZeppelinContext` - wich is used for dynamic forms, in case py4j is installed on the system for things like

      z.select("Choose a letter", ([a,"a"], [b,"b"], [c,"c"]))
      

      In latter case Python interpreter becomes not useable - any line results in

      x = 1
      
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      AttributeError: 'Py4jZeppelinContext' object has no attribute '_displayhook'
      

        Issue Links

          Activity

          Hide
          bzz Alexander Bezzubov added a comment -

          Alex Goodman if you have some time, could you please take a look?

          Show
          bzz Alexander Bezzubov added a comment - Alex Goodman if you have some time, could you please take a look?
          Hide
          agoodman Alex Goodman added a comment - - edited

          Alexander Bezzubov Can't believe I missed this. The Py4jZeppelinContext constructor needs an additional line which calls the superclass constructor in-order to set the _displayhook attribute. This is a very easy fix.

          Show
          agoodman Alex Goodman added a comment - - edited Alexander Bezzubov Can't believe I missed this. The Py4jZeppelinContext constructor needs an additional line which calls the superclass constructor in-order to set the _displayhook attribute. This is a very easy fix.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user agoodm opened a pull request:

          https://github.com/apache/zeppelin/pull/1626

          ZEPPELIN-1655 Dynamic forms in Python interpreter do not work

              1. What is this PR for?
                After #1534 , Dynamic Forms were no longer working in the python interpreter. This is because the `Py4jZeppelinContext` constructor did not initialize the `_displayhook` which is always called on post-execute.
              1. What type of PR is it?
                Bug Fix
              1. What is the Jira issue?
                ZEPPELIN-1655(https://issues.apache.org/jira/browse/ZEPPELIN-1655)
              1. How should this be tested?
                Run the following `%python` paragraph, being sure that Py4j is installed:
                ```python
                %python
                a, b, c, = (1, 2, 3)
                z.select("Choose a letter", ([a,"a"], [b,"b"], [c,"c"] ))
                ```
              1. Questions:
          • Does the licenses files need update? No
          • Is there breaking changes for older versions? No
          • Does this needs documentation? No

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/agoodm/zeppelin ZEPPELIN-1655

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/zeppelin/pull/1626.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1626


          commit 2e4ee2dd41314cb2f1f7c999484397d0f0b791b3
          Author: Alex Goodman <agoodm@users.noreply.github.com>
          Date: 2016-11-13T18:10:50Z

          Make sure _displayhook is initialized in Py4jZeppelinContext


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user agoodm opened a pull request: https://github.com/apache/zeppelin/pull/1626 ZEPPELIN-1655 Dynamic forms in Python interpreter do not work What is this PR for? After #1534 , Dynamic Forms were no longer working in the python interpreter. This is because the `Py4jZeppelinContext` constructor did not initialize the `_displayhook` which is always called on post-execute. What type of PR is it? Bug Fix What is the Jira issue? ZEPPELIN-1655 ( https://issues.apache.org/jira/browse/ZEPPELIN-1655 ) How should this be tested? Run the following `%python` paragraph, being sure that Py4j is installed: ```python %python a, b, c, = (1, 2, 3) z.select("Choose a letter", ( [a,"a"] , [b,"b"] , [c,"c"] )) ``` Questions: Does the licenses files need update? No Is there breaking changes for older versions? No Does this needs documentation? No You can merge this pull request into a Git repository by running: $ git pull https://github.com/agoodm/zeppelin ZEPPELIN-1655 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/1626.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1626 commit 2e4ee2dd41314cb2f1f7c999484397d0f0b791b3 Author: Alex Goodman <agoodm@users.noreply.github.com> Date: 2016-11-13T18:10:50Z Make sure _displayhook is initialized in Py4jZeppelinContext
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user agoodm commented on the issue:

          https://github.com/apache/zeppelin/pull/1626

          Here you go @bzz !

          Show
          githubbot ASF GitHub Bot added a comment - Github user agoodm commented on the issue: https://github.com/apache/zeppelin/pull/1626 Here you go @bzz !
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user bzz commented on the issue:

          https://github.com/apache/zeppelin/pull/1626

          Thank you for the prompt update @agoodm

          CI failure on Selenium test profile seem un-related:

          ```
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 129.268 sec - in org.apache.zeppelin.integration.ZeppelinIT

          Results :

          Failed tests:
          AuthenticationIT.startUp:95 null

          Tests run: 17, Failures: 1, Errors: 0, Skipped: 0
          ```

          Looks gearat to me, merging to master as a hotfix, if there is no further discussion.

          Show
          githubbot ASF GitHub Bot added a comment - Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1626 Thank you for the prompt update @agoodm CI failure on Selenium test profile seem un-related: ``` Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 129.268 sec - in org.apache.zeppelin.integration.ZeppelinIT Results : Failed tests: AuthenticationIT.startUp:95 null Tests run: 17, Failures: 1, Errors: 0, Skipped: 0 ``` Looks gearat to me, merging to master as a hotfix, if there is no further discussion.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/zeppelin/pull/1626

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/zeppelin/pull/1626
          Hide
          bzz Alexander Bezzubov added a comment - - edited

          Alex Goodman I must be doing something wrong, but after #1626 I can not run any python paragraph (with, or without py4j installed):

          %python
          x = 1
          
          Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
          NameError: name 'z' is not defined
          
          Show
          bzz Alexander Bezzubov added a comment - - edited Alex Goodman I must be doing something wrong, but after #1626 I can not run any python paragraph (with, or without py4j installed): %python x = 1 Traceback (most recent call last): File "<stdin>" , line 1, in <module> NameError: name 'z' is not defined
          Hide
          agoodman Alex Goodman added a comment -

          Alexander Bezzubov Hmm, I can run paragraphs like that just fine. Have you tried this after building Zeppelin from a clean repository (as in, git clone the master branch to a new empty directory)?

          My current configuration:
          Python: 2.7.12
          Maven: 3.3.9
          Java: 1.7.0_80
          OS: Linux Mint

          I can test this on a different computer tomorrow but either way this all sounds very strange.

          Show
          agoodman Alex Goodman added a comment - Alexander Bezzubov Hmm, I can run paragraphs like that just fine. Have you tried this after building Zeppelin from a clean repository (as in, git clone the master branch to a new empty directory)? My current configuration: Python: 2.7.12 Maven: 3.3.9 Java: 1.7.0_80 OS: Linux Mint I can test this on a different computer tomorrow but either way this all sounds very strange.
          Hide
          bzz Alexander Bezzubov added a comment - - edited

          Thank you for double-checking - I tried everything one more time as well, with a clean clone.

          Python: Python 2.7.12
          Apache Maven 3.3.9
          Java version: 1.8.0_111
          OS: Ubuntu 16.04

          Built with mvn clean package -DskipTests -P beam -pl !zeppelin-distribution

          x=1
          
          Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
          NameError: name 'z' is not defined
          

          Logs

          DEBUG [2016-11-13 23:05:55,422] ({pool-2-thread-2} PythonInterpreter.java[sendCommandToPython]:230) - Sending :
          x = 1
          z._displayhook()
          DEBUG [2016-11-13 23:05:55,422] ({pool-2-thread-2} PythonProcess.java[sendAndGetResult]:93) - Read line from python shell : Traceback (most recent call last):
          DEBUG [2016-11-13 23:05:55,422] ({pool-2-thread-2} PythonProcess.java[sendAndGetResult]:93) - Read line from python shell :   File "<stdin>", line 1, in <module>
          DEBUG [2016-11-13 23:05:55,422] ({pool-2-thread-2} PythonProcess.java[sendAndGetResult]:93) - Read line from python shell : NameError: name 'z' is not defined
          DEBUG [2016-11-13 23:05:55,422] ({pool-2-thread-2} PythonInterpreter.java[sendCommandToPython]:236) - Got :
          Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
          NameError: name 'z' is not defined
          

          Very strange indeed...

          Show
          bzz Alexander Bezzubov added a comment - - edited Thank you for double-checking - I tried everything one more time as well, with a clean clone. Python: Python 2.7.12 Apache Maven 3.3.9 Java version: 1.8.0_111 OS: Ubuntu 16.04 Built with mvn clean package -DskipTests -P beam -pl !zeppelin-distribution x=1 Traceback (most recent call last): File "<stdin>" , line 1, in <module> NameError: name 'z' is not defined Logs DEBUG [2016-11-13 23:05:55,422] ({pool-2-thread-2} PythonInterpreter.java[sendCommandToPython]:230) - Sending : x = 1 z._displayhook() DEBUG [2016-11-13 23:05:55,422] ({pool-2-thread-2} PythonProcess.java[sendAndGetResult]:93) - Read line from python shell : Traceback (most recent call last): DEBUG [2016-11-13 23:05:55,422] ({pool-2-thread-2} PythonProcess.java[sendAndGetResult]:93) - Read line from python shell : File "<stdin>" , line 1, in <module> DEBUG [2016-11-13 23:05:55,422] ({pool-2-thread-2} PythonProcess.java[sendAndGetResult]:93) - Read line from python shell : NameError: name 'z' is not defined DEBUG [2016-11-13 23:05:55,422] ({pool-2-thread-2} PythonInterpreter.java[sendCommandToPython]:236) - Got : Traceback (most recent call last): File "<stdin>" , line 1, in <module> NameError: name 'z' is not defined Very strange indeed...
          Hide
          agoodman Alex Goodman added a comment -

          I recall having a problem similar to this one in the past, since I know that the current python interpreter impl is very picky about whitespace and indentation (to an incorrect degree, as we both know). What's strange in this case is that you are having this problem but I am not. Maybe you should try adding some extra logging in PythonInterpreter.java which prints out the exact text being loaded bootstrap.py into the interpreter on your end.

          Show
          agoodman Alex Goodman added a comment - I recall having a problem similar to this one in the past, since I know that the current python interpreter impl is very picky about whitespace and indentation (to an incorrect degree, as we both know). What's strange in this case is that you are having this problem but I am not. Maybe you should try adding some extra logging in PythonInterpreter.java which prints out the exact text being loaded bootstrap.py into the interpreter on your end.
          Hide
          agoodman Alex Goodman added a comment -

          Alexander Bezzubov I think I have found the source of the problem. I'll submit a PR shortly for you to test, though it will be under ZEPPELIN-1656.

          Show
          agoodman Alex Goodman added a comment - Alexander Bezzubov I think I have found the source of the problem. I'll submit a PR shortly for you to test, though it will be under ZEPPELIN-1656 .
          Hide
          bzz Alexander Bezzubov added a comment - - edited

          Thank you! I did enabled DEBUG logging, and here is something else found in the logs, which was never shown in GUI:

            File "<stdin>", line 1, in <module>
            File "<stdin>", line 11, in __init__
            File "<stdin>", line 108, in _setup_matplotlib
          UnboundLocalError: local variable 'matplotlib' referenced before assignment
          

          Which made me thing that on a new machine, there was no matplotlib installed!

          And indeed, after installing it - I'm able to run a paragraphs again.

          How do you think, would it make sense to add some explicit dependency checks for matolotlib at the beginning, that can fail in a user-friendly manner like we did in `py4j` - https://github.com/apache/zeppelin/blob/aded8681bb06e2ca897253820640b153afa3c468/python/src/main/resources/bootstrap.py#L116 or is there a better way to verify the dependencies ?

          Show
          bzz Alexander Bezzubov added a comment - - edited Thank you! I did enabled DEBUG logging, and here is something else found in the logs, which was never shown in GUI: File "<stdin>" , line 1, in <module> File "<stdin>" , line 11, in __init__ File "<stdin>" , line 108, in _setup_matplotlib UnboundLocalError: local variable 'matplotlib' referenced before assignment Which made me thing that on a new machine, there was no matplotlib installed! And indeed, after installing it - I'm able to run a paragraphs again. How do you think, would it make sense to add some explicit dependency checks for matolotlib at the beginning, that can fail in a user-friendly manner like we did in `py4j` - https://github.com/apache/zeppelin/blob/aded8681bb06e2ca897253820640b153afa3c468/python/src/main/resources/bootstrap.py#L116 or is there a better way to verify the dependencies ?
          Hide
          agoodman Alex Goodman added a comment -

          That's actually not supposed to happen, the `_setup_matplotib()` method should simply exit when matplotlib cannot be imported. I think the approach of just checking for imports at the python level is fine as the behavior is correct in pyspark but apparently for python I used a pass statement instead of return. A very silly mistake indeed. I'll fix this right now.

          Show
          agoodman Alex Goodman added a comment - That's actually not supposed to happen, the `_setup_matplotib()` method should simply exit when matplotlib cannot be imported. I think the approach of just checking for imports at the python level is fine as the behavior is correct in pyspark but apparently for python I used a pass statement instead of return . A very silly mistake indeed. I'll fix this right now.

            People

            • Assignee:
              agoodman Alex Goodman
              Reporter:
              bzz Alexander Bezzubov
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development