Pig
  1. Pig
  2. PIG-1280

Add a pig-script-id to the JobConf of all jobs run in a pig-script

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: impl
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      A new Pig property ("pig.script.info.enabled") is added to the pig-default.properties file that allows Pig to add the user script, input dirs, and other script related settings to all MR jobs produced by the script (and so being persisted in the job xmls). The default value is true. To turn off this feature, set the value to false.
      Show
      A new Pig property ("pig.script.info.enabled") is added to the pig-default.properties file that allows Pig to add the user script, input dirs, and other script related settings to all MR jobs produced by the script (and so being persisted in the job xmls). The default value is true. To turn off this feature, set the value to false.

      Description

      It would be very useful for tools like gridmix if pig could add a 'pig-script-id' to all Map-Reduce jobs spawned by a single pig-script. Potentially we could use this to re-construct the DAG of jobs in gridmix and so on.

      1. PIG-1280.patch
        18 kB
        Richard Ding
      2. PIG-1280.patch
        13 kB
        Richard Ding

        Issue Links

          Activity

          Hide
          Richard Ding added a comment -

          The hudson servers are down. So I run the unit tests locally and they are successful. I'm going to check in the patch.

          Show
          Richard Ding added a comment - The hudson servers are down. So I run the unit tests locally and they are successful. I'm going to check in the patch.
          Hide
          Daniel Dai added a comment -

          With PIG-1381 checked in, we need to add config entry into pig-default.properties instead of pig.properties. Note this change when commit.

          Show
          Daniel Dai added a comment - With PIG-1381 checked in, we need to add config entry into pig-default.properties instead of pig.properties. Note this change when commit.
          Hide
          Daniel Dai added a comment -

          +1 for the new patch if tests pass.

          Show
          Daniel Dai added a comment - +1 for the new patch if tests pass.
          Hide
          Richard Ding added a comment -

          New patch adding a Pig property that allows user to turn off this feature.

          Show
          Richard Ding added a comment - New patch adding a Pig property that allows user to turn off this feature.
          Hide
          Daniel Dai added a comment -

          Patch looks good. Do we need a switch in pig.property file to turn this off? I think these are optional information we can expose some control to users.

          Show
          Daniel Dai added a comment - Patch looks good. Do we need a switch in pig.property file to turn this off? I think these are optional information we can expose some control to users.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12444018/PIG-1280.patch
          against trunk revision 941976.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/313/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/313/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/313/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444018/PIG-1280.patch against trunk revision 941976. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/313/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/313/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/313/console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12444018/PIG-1280.patch
          against trunk revision 941976.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/311/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/311/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/311/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444018/PIG-1280.patch against trunk revision 941976. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/311/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/311/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/311/console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12444018/PIG-1280.patch
          against trunk revision 941976.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/310/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/310/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/310/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444018/PIG-1280.patch against trunk revision 941976. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/310/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/310/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/310/console This message is automatically generated.
          Hide
          Richard Ding added a comment -

          Patch to add Pig properties to the MR jobs.

          Show
          Richard Ding added a comment - Patch to add Pig properties to the MR jobs.
          Hide
          Richard Ding added a comment -

          There have been several similar requests (on adding new Pig properties to MR job) since this Jira was filed. Here are a compilation of those properties:

          • pig.script.id
          • pig.script (the Pig script generates this job)
          • pig.launcher.host (the host/IP of the machine on which the Pig script is executed)
          • pig.command.line (the Pig command line arguments of this script)
          • pig.input.dirs (comma separated input directory list of this job)
          • pig.output.dirs (comma separated output directory list of this job)
          • pig.version
          Show
          Richard Ding added a comment - There have been several similar requests (on adding new Pig properties to MR job) since this Jira was filed. Here are a compilation of those properties: pig.script.id pig.script (the Pig script generates this job) pig.launcher.host (the host/IP of the machine on which the Pig script is executed) pig.command.line (the Pig command line arguments of this script) pig.input.dirs (comma separated input directory list of this job) pig.output.dirs (comma separated output directory list of this job) pig.version

            People

            • Assignee:
              Richard Ding
              Reporter:
              Arun C Murthy
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development