[PIG-3288] Kill jobs if the number of output files is over a configurable limit

• Type: Wish
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Assignee: Cheolsoo Park
• Reporter: Cheolsoo Park
• Created: 22/Apr/13 21:06
• Updated: 07/Sep/14 03:34
• Resolved: 07/Sep/14 03:34

Description

I ran into a situation where a Pig job tried to create too many files on hdfs and overloaded NN. To prevent such events, it would be nice if we could set a upper limit on the number of files that a Pig job can create.

In fact, Hive has a property called "hive.exec.max.created.files". The idea is that each mapper/reducer increases a counter every time when they create files. Then, MRLauncher periodically checks whether the number of created files so far has exceeded the upper limit. If so, we kill running jobs and exit.