Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4059 Pig on Spark
  3. PIG-4920

Fail to use Javascript UDF in spark yarn client mode

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:
      None

      Description

      udf.pig

      register '/home/zly/prj/oss/merge.pig/pig/bin/udf.js' using javascript as myfuncs;
      A = load './passwd' as (a0:chararray, a1:chararray);
      B = foreach A generate myfuncs.helloworld();
      store B into './udf.out';
      

      udf.js

      helloworld.outputSchema = "word:chararray";
      function helloworld() {
          return 'Hello, World';
      }
          
      complex.outputSchema = "word:chararray";
      function complex(word){
          return {word:word};
      }
      

      run udf.pig in spark local mode(export SPARK_MASTER="local"), it successfully.
      run udf.pig in spark yarn client mode(export SPARK_MASTER="yarn-client"), it fails and error message like following:

      Caused by: java.lang.reflect.InvocationTargetException
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
              at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:744)
              ... 84 more
      Caused by: java.lang.ExceptionInInitializerError
              at org.apache.pig.scripting.js.JsScriptEngine.getInstance(JsScriptEngine.java:87)
              at org.apache.pig.scripting.js.JsFunction.<init>(JsFunction.java:173)
              ... 89 more
      Caused by: java.lang.IllegalStateException: could not get script path from UDFContext
              at org.apache.pig.scripting.js.JsScriptEngine$Holder.<clinit>(JsScriptEngine.java:69)
              ... 91 more
      

        Attachments

        1. PIG-4920.patch
          5 kB
          liyunzhang
        2. PIG-4920_2.patch
          7 kB
          liyunzhang
        3. PIG-4920_3.patch
          7 kB
          liyunzhang
        4. PIG-4920_4.patch
          15 kB
          liyunzhang
        5. PIG-4920_5.patch
          16 kB
          liyunzhang
        6. PIG-4920_6.patch
          17 kB
          liyunzhang

          Issue Links

            Activity

              People

              • Assignee:
                kellyzly liyunzhang
                Reporter:
                kellyzly liyunzhang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: