Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
None
-
None
-
None
-
None
-
Pig Local
Description
This is to separate additional improvements and original requirement on PIG-129 issue.
Pasting comments of Pi Song.
"I think the concept of multi-dir temp file creator (LocalDirAllocator in Hadoop) should be adopted to Pig. What it does is:-
- You can set up a set of tmp file dirs in configuration (They can be on different physical drives so you can utilize more disk space)
- When a temp file is being created, the system will probe the given temp dirs in round-robin fashion
- For a selected temp dir, if it exists and you have permission to write, temp file will be created
- For a selected temp dir, it it doesn't exist or you don't have permission to write, the temp dir will be kept in the black list, thus not being used later on.
- For the next temp file, move on to the next temp dir
"