Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3288

Kill jobs if the number of output files is over a configurable limit

    Details

    • Type: Wish
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I ran into a situation where a Pig job tried to create too many files on hdfs and overloaded NN. To prevent such events, it would be nice if we could set a upper limit on the number of files that a Pig job can create.

      In fact, Hive has a property called "hive.exec.max.created.files". The idea is that each mapper/reducer increases a counter every time when they create files. Then, MRLauncher periodically checks whether the number of created files so far has exceeded the upper limit. If so, we kill running jobs and exit.

        Attachments

        1. PIG-3288-5.patch
          13 kB
          Cheolsoo Park
        2. PIG-3288-4.patch
          11 kB
          Cheolsoo Park
        3. PIG-3288-3.patch
          11 kB
          Cheolsoo Park
        4. PIG-3288-2.patch
          11 kB
          Cheolsoo Park
        5. PIG-3288.patch
          9 kB
          Cheolsoo Park

          Activity

            People

            • Assignee:
              cheolsoo Cheolsoo Park
              Reporter:
              cheolsoo Cheolsoo Park
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: