Pig
  1. Pig
  2. PIG-1478

Add progress notification listener to PigRunner API

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      PIG-1333 added PigRunner API to allow Pig users and tools to get a status/stats object back after executing a Pig script. The new API, however, is synchronous (blocking). It's known that a Pig script can spawn tens (even hundreds) MR jobs and take hours to complete. Therefore it'll be nice to give progress feedback to the callers during the execution.

      The proposal is to add an optional parameter to the API:

      public abstract class PigRunner {
          public static PigStats run(String[] args, PigProgressNotificationListener listener) {...}
      }
      

      The new listener is defined as following:

      package org.apache.pig.tools.pigstats;
      
      public interface PigProgressNotificationListener extends java.util.EventListener {
          // just before the launch of MR jobs for the script
          public void LaunchStartedNotification(int numJobsToLaunch);
          // number of jobs submitted in a batch
          public void jobsSubmittedNotification(int numJobsSubmitted);
          // a job is started
          public void jobStartedNotification(String assignedJobId);
          // a job is completed successfully
          public void jobFinishedNotification(JobStats jobStats);
          // a job is failed
          public void jobFailedNotification(JobStats jobStats);
          // a user output is completed successfully
          public void outputCompletedNotification(OutputStats outputStats);
          // updates the progress as percentage
          public void progressUpdatedNotification(int progress);
          // the script execution is done
          public void launchCompletedNotification(int numJobsSucceeded);
      }
      

      Any thoughts?

      1. PIG-1478.patch
        19 kB
        Richard Ding

        Activity

        Richard Ding created issue -
        Richard Ding made changes -
        Field Original Value New Value
        Attachment PIG-1478.patch [ 12448532 ]
        Richard Ding made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Richard Ding made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Olga Natkovich made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Richard Ding
            Reporter:
            Richard Ding
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development