Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2456

spark action can not find pyspark module

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 4.1.0
    • Fix Version/s: None
    • Component/s: action, client, core
    • Labels:
      None
    • Environment:

      Ubuntu 14.04.3

      Description

      I hava a spark script written in pyspark and I want to submit it via oozie spark action.
      something like this:

        <action name="myapp">
            <spark xmlns="uri:oozie:spark-action:0.1">
                <job-tracker>${job_tracker}</job-tracker>
                <name-node>${name_node}</name-node>
                <master>local[*]</master>
                <name>myapp</name>
                <jar>${my_script}</jar>
                <spark-opts>--executor-memory 4G --num-executors 4</spark-opts>
                <arg>${arg1}</arg>
            </spark>
            <ok to="hive_import"/>
            <error to="send_email"/>
        </action>
      

      The script imports pyspark module:

      Unable to find source-code formatter for language: text. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      #!/usr/bin/spark-submit
      from pyspark import SparkContext
      from pyspark import SparkFiles
      sc = SparkContext()
      

      However, the oozie will throw the " Can not import pyspark module" exception.
      This happens when I upgrade to CDH 5.5.1 from CDH 5.4.6.
      The workaround would be using the shell action, but I think the spark action is better to describe the spark task.
      Any suggestion?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                alec.tu Ming Hsuan Tu
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: