Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-1967

Passing Z variables to Shell and SQL Interpreters

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0, 0.8.0
    • Fix Version/s: 0.8.0
    • Labels:
      None

      Description

      It's possible to exchange variables between Scala and Spark
      through z.put and z.get.

      How to pass a variable to %sh and %sql?

      In Jupyter it's possible to do for example as

        ! hadoop fs -put {localfile} {hdfsfile}
      

      where localfile and and hdfsfile are Python variables.

      Can't find any references for something similar in Shell Interpreter
      https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/shell.html

      In many notebooks we have to pass small variabels
      from Zeppelin notes to external scripts as parameters.

      It would be awesome to have something like

      %sh
      /path/to/script --param8={var1} --param9={var2}
      

      where var1 and var2 would be implied to be fetched as z.get('var1')
      and z.get('var2') respectively.

      Or similarly for %sql :

      %sql
      create table dwh.table_{year} stores as parquet
      as 
      select * from spark_df1 where year = {year}
      

      We miss a lot global variables for %sql and %sh so that a Zeppelin note can be used as a single parametrized
      orchestration for a whole workflow.

      Escaping is done similarly like in Jupyter.
      With double curly braces –

        {{ will be transformed as single left curly brace - {
        }} will be escaped to single right curly brace - }
      

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Tagar Ruslan Dautkhanov

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment