[PIG-4913] Reduce jython function initiation during compilation

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Assignee: Ádám Szita
• Reporter: Rohini Palaniswamy
• Created: 31/May/16 17:43
• Updated: 07/Feb/17 13:25

Description

While investigating PIG-4908, saw that ScriptEngine.getScriptAsStream was invoked way too many times during compilation phase for a simple script.

sleep.py

#!/usr/bin/python

import time;

@outputSchema("sltime:int")
def sleep(num):
    if num == 1:
        print "Sleeping for %d minutes" % num;
        time.sleep(num * 60);
    return num;

sleep.pig

register 'sleep.py' using jython;

A = LOAD '/tmp/sleepdata' as (f1:int);
B = FOREACH A generate $0, sleep($0);
STORE B into '/tmp/tezout';