[PIG-1752] UDFs should be able to indicate files to load in the distributed cache

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Component/s: impl

Release Note

We add a new method to EvalFunc:
public List<String> getCacheFiles();

User can override this method to return a list of hdfs files need to shipped to distributed cache. Inside EvalFunc, user can assume these files are already exist in distributed cache.

For example:
public class Udfcachetest extends EvalFunc<String> {

    public String exec(Tuple input) throws IOException {
        FileReader fr = new FileReader("./smallfile");
        BufferedReader d = new BufferedReader(fr);
        return d.readLine();
    }

    public List<String> getCacheFiles() {
        List<String> list = new ArrayList<String>(1);
        list.add("/user/pig/tests/data/small#smallfile");
        return list;
    }
}

a = load '1.txt';
b = foreach a generate Udfcachetest(*);
dump b;

• Assignee: Alan Gates
• Reporter: Alan Gates
• Created: 03/Dec/10 00:49
• Updated: 04/Aug/11 00:35
• Resolved: 06/Dec/10 21:45

Description

Currently there is no way for a UDF to load a file into the distributed cache.

Issue Links

• relates to: PIG-1337 Need a way to pass distributed cache configuration information to hadoop backend in Pig's LoadFunc