Pig
  1. Pig
  2. PIG-1794

Javascript support for Pig embedding and UDFs in scripting languages

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.9.0
    • Component/s: impl
    • Labels:
      None
    • Patch Info:
      Patch Available
    • Hadoop Flags:
      Reviewed

      Description

      The attached patch proposes a javascript implementation for Pig embedding and UDFs in scripting languages.
      It is similar to the Jython implementation and uses Rhino provided in the JDK.
      some differences:

      • output schema is provided by: <functionName>.outSchema="<schema>" as javascript does not have annotations or decorators but functions are first class objects
      • tuples are converted to objects using the input schema (the other way around using the output schema)

      The attached patch is not final yet. In particular it lacks unit tests.
      See test/org/apache/pig/test/data/tc.js for the "transitive closure" example

      See the following JIRAs for more context:
      https://issues.apache.org/jira/browse/PIG-928
      https://issues.apache.org/jira/browse/PIG-1479

      1. jsScripting-6.patch
        66 kB
        Julien Le Dem
      2. jsScripting-5.patch
        64 kB
        Julien Le Dem
      3. jsScripting-4.patch
        64 kB
        Julien Le Dem
      4. jsScripting-3.patch
        62 kB
        Julien Le Dem
      5. jsScripting-2.patch
        58 kB
        Julien Le Dem
      6. jsScripting.patch
        44 kB
        Julien Le Dem

        Issue Links

          Activity

          Hide
          Julien Le Dem added a comment -

          I've limited modifications to the existing framework as much as possible.
          However some changes where necessary. They are backward compatible.

          Show
          Julien Le Dem added a comment - I've limited modifications to the existing framework as much as possible. However some changes where necessary. They are backward compatible.
          Hide
          Julien Le Dem added a comment -
          Show
          Julien Le Dem added a comment - see review: https://reviews.apache.org/r/321/
          Hide
          David Ciemiewicz added a comment -

          Julien,

          This is extremely interesting to me. Do you have sample code showing Pig embedded in JavaScript and/or Pig UDFs written in JavaScript?

          Thanks!

          Show
          David Ciemiewicz added a comment - Julien, This is extremely interesting to me. Do you have sample code showing Pig embedded in JavaScript and/or Pig UDFs written in JavaScript? Thanks!
          Hide
          Julien Le Dem added a comment -

          Hi Ciemo,
          There's an example as part of the patch here:
          https://reviews.apache.org/r/321/diff/#12
          Julien

          Show
          Julien Le Dem added a comment - Hi Ciemo, There's an example as part of the patch here: https://reviews.apache.org/r/321/diff/#12 Julien
          Hide
          David Ciemiewicz added a comment -

          Julien ... very nice!

          Are Accumulator and Algebraic functions supported?

          Show
          David Ciemiewicz added a comment - Julien ... very nice! Are Accumulator and Algebraic functions supported?
          Hide
          David Ciemiewicz added a comment -

          Also, is an "include" mechanism supported so that we can have a "library" of JavaScript functions that we reuse from script to script?

          Show
          David Ciemiewicz added a comment - Also, is an "include" mechanism supported so that we can have a "library" of JavaScript functions that we reuse from script to script?
          Hide
          Julien Le Dem added a comment -

          Accumulator and Algebraic are planned but not there yet.
          Include should work as long as the included scripts are in the class path but I have not tried it.

          Show
          Julien Le Dem added a comment - Accumulator and Algebraic are planned but not there yet. Include should work as long as the included scripts are in the class path but I have not tried it.
          Hide
          Richard Ding added a comment -

          Review comments are here https://reviews.apache.org/r/321/.

          Since the review board hasn't been linked with Jira, please upload the new patch to the jira.

          As for the 'include' statement, a related jira is PIG-1824 where the idea is to add SHIP clause so Pig would ship 'import/include' scripts to the backend.

          Show
          Richard Ding added a comment - Review comments are here https://reviews.apache.org/r/321/ . Since the review board hasn't been linked with Jira, please upload the new patch to the jira. As for the 'include' statement, a related jira is PIG-1824 where the idea is to add SHIP clause so Pig would ship 'import/include' scripts to the backend.
          Hide
          Julien Le Dem added a comment -
          Show
          Julien Le Dem added a comment - Patch reviewed in https://reviews.apache.org/r/321/
          Hide
          Julien Le Dem added a comment -

          jsScript-3.patch: updated patch to work with the latest modifications in trunk.

          Show
          Julien Le Dem added a comment - jsScript-3.patch: updated patch to work with the latest modifications in trunk.
          Hide
          Richard Ding added a comment -

          Output of test-patch:

               [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 10 new or modified tests.
               [exec] 
               [exec]     -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.
               [exec] 
               [exec]     -1 javac.  The applied patch generated 940 javac compiler warnings (more than the trunk's current 939 warnings).
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     -1 release audit.  The applied patch generated 523 release audit warnings (more than the trunk's current 521 warnings).
               [exec] 
          
          
          Show
          Richard Ding added a comment - Output of test-patch: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 10 new or modified tests. [exec] [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages. [exec] [exec] -1 javac. The applied patch generated 940 javac compiler warnings (more than the trunk's current 939 warnings). [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 523 release audit warnings (more than the trunk's current 521 warnings). [exec]
          Hide
          Julien Le Dem added a comment -

          jsScripting-4.patch addresses the warnings found in previous patch.

          Show
          Julien Le Dem added a comment - jsScripting-4.patch addresses the warnings found in previous patch.
          Hide
          Richard Ding added a comment -

          Output of test-patch:

               [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 10 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec] 
          
          Show
          Richard Ding added a comment - Output of test-patch: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 10 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec]
          Hide
          Julien Le Dem added a comment -

          jsScripting-5.patch fixes a bug for shipping js.jar with the job. Updated CHANGES.txt
          Tested on an actual grid.

          Show
          Julien Le Dem added a comment - jsScripting-5.patch fixes a bug for shipping js.jar with the job. Updated CHANGES.txt Tested on an actual grid.
          Hide
          Julien Le Dem added a comment -

          jsScripting-6.patch addresses an intermittent bug related to how Rhino calls Java methods.

          Show
          Julien Le Dem added a comment - jsScripting-6.patch addresses an intermittent bug related to how Rhino calls Java methods.
          Hide
          Richard Ding added a comment -

          +1

          Show
          Richard Ding added a comment - +1
          Hide
          Julien Le Dem added a comment -

          jsScript-6.patch was commited to TRUNK

          Show
          Julien Le Dem added a comment - jsScript-6.patch was commited to TRUNK

            People

            • Assignee:
              Julien Le Dem
              Reporter:
              Julien Le Dem
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development