Pig
  1. Pig
  2. PIG-3288

Kill jobs if the number of output files is over a configurable limit

    Details

    • Type: Wish Wish
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I ran into a situation where a Pig job tried to create too many files on hdfs and overloaded NN. To prevent such events, it would be nice if we could set a upper limit on the number of files that a Pig job can create.

      In fact, Hive has a property called "hive.exec.max.created.files". The idea is that each mapper/reducer increases a counter every time when they create files. Then, MRLauncher periodically checks whether the number of created files so far has exceeded the upper limit. If so, we kill running jobs and exit.

      1. PIG-3288-5.patch
        13 kB
        Cheolsoo Park
      2. PIG-3288-4.patch
        11 kB
        Cheolsoo Park
      3. PIG-3288-3.patch
        11 kB
        Cheolsoo Park
      4. PIG-3288-2.patch
        11 kB
        Cheolsoo Park
      5. PIG-3288.patch
        9 kB
        Cheolsoo Park

        Activity

        Cheolsoo Park created issue -
        Cheolsoo Park made changes -
        Field Original Value New Value
        Attachment PIG-3288.patch [ 12579905 ]
        Cheolsoo Park made changes -
        Attachment PIG-3288-2.patch [ 12580385 ]
        Cheolsoo Park made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Cheolsoo Park made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Cheolsoo Park made changes -
        Attachment PIG-3288-3.patch [ 12585676 ]
        Cheolsoo Park made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Cheolsoo Park made changes -
        Attachment PIG-3288-4.patch [ 12587091 ]
        Cheolsoo Park made changes -
        Attachment PIG-3288-5.patch [ 12589295 ]
        Cheolsoo Park made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Cheolsoo Park made changes -
        Fix Version/s 0.12 [ 12323380 ]

          People

          • Assignee:
            Cheolsoo Park
            Reporter:
            Cheolsoo Park
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development