[PIG-2925] Extremely long JobConf values should not be added to Streaming environment

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Assignee: Dmitriy V. Ryaboy
• Reporter: Dmitriy V. Ryaboy
• Created: 19/Sep/12 23:33
• Updated: 15/Oct/12 20:43
• Resolved: 15/Oct/12 20:43

Description

PIG-2900 made Pig match Hadoop's practice of making all JobConf properties available as environment variables for pig streaming.

As it happens, ProcessBuilder creates a long string with all environment variables and their values on it, then runs it; having some of the weightier properties like serialized plans in the environment can make the total command length exceed OS limits.

We propose adding a blacklist of properties which should not be added to streaming environment.

Issue Links

• is superceded by: PIG-2971 Add new parameter to specify the streaming environment