Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-610

Oozie system share lib should have jars per action type

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).

      As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.

      This creates a few issues:

      • The classpath for an action grows significantly
      • For a given action only a portion of the classpath is useful, the rest is dead code
      • As more actions are added, the chances of conflicting dependencies grows.

      As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.

      Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.

      This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:

      share/lib/pig/*
      share/lib/streaming/*
      share/lib/hive/*
      share/lib/sqoop/*

      Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.

      This would move the resolution for the Oozie share lib from the submit-command to the action-executor.

      Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.

      Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.

        Issue Links

          Activity

          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2909/
          -----------------------------------------------------------

          Review request for oozie.

          Summary
          -------

          breaks the Oozie sharelib into one subdir per action type

          This addresses bug OOZIE-610.
          https://issues.apache.org/jira/browse/OOZIE-610

          Diffs


          /trunk/core/pom.xml 1204834
          /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1204834
          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1204834
          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1204834
          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1204834
          /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1204834
          /trunk/docs/src/site/twiki/DG_Examples.twiki 1204834
          /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1204834
          /trunk/examples/pom.xml 1204834
          /trunk/examples/src/main/apps/custom-main/job.properties 1204834
          /trunk/examples/src/main/apps/demo/job.properties 1204834
          /trunk/examples/src/main/apps/hadoop-el/job.properties 1204834
          /trunk/examples/src/main/apps/java-main/job.properties 1204834
          /trunk/examples/src/main/apps/pig/job.properties 1204834
          /trunk/examples/src/main/apps/streaming/job.properties 1204834
          /trunk/pom.xml 1204834
          /trunk/sharelib/pig/pom.xml PRE-CREATION
          /trunk/sharelib/pom.xml 1204834
          /trunk/sharelib/streaming/pom.xml PRE-CREATION
          /trunk/src/main/assemblies/distro.xml 1204834
          /trunk/src/main/assemblies/examples.xml 1204834
          /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
          /trunk/src/main/assemblies/sharelib.xml 1204834

          Diff: https://reviews.apache.org/r/2909/diff

          Testing
          -------

          built, installed and run pig and streaming examples

          Thanks,

          Alejandro

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/ ----------------------------------------------------------- Review request for oozie. Summary ------- breaks the Oozie sharelib into one subdir per action type This addresses bug OOZIE-610 . https://issues.apache.org/jira/browse/OOZIE-610 Diffs /trunk/core/pom.xml 1204834 /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1204834 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1204834 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1204834 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1204834 /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1204834 /trunk/docs/src/site/twiki/DG_Examples.twiki 1204834 /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1204834 /trunk/examples/pom.xml 1204834 /trunk/examples/src/main/apps/custom-main/job.properties 1204834 /trunk/examples/src/main/apps/demo/job.properties 1204834 /trunk/examples/src/main/apps/hadoop-el/job.properties 1204834 /trunk/examples/src/main/apps/java-main/job.properties 1204834 /trunk/examples/src/main/apps/pig/job.properties 1204834 /trunk/examples/src/main/apps/streaming/job.properties 1204834 /trunk/pom.xml 1204834 /trunk/sharelib/pig/pom.xml PRE-CREATION /trunk/sharelib/pom.xml 1204834 /trunk/sharelib/streaming/pom.xml PRE-CREATION /trunk/src/main/assemblies/distro.xml 1204834 /trunk/src/main/assemblies/examples.xml 1204834 /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION /trunk/src/main/assemblies/sharelib.xml 1204834 Diff: https://reviews.apache.org/r/2909/diff Testing ------- built, installed and run pig and streaming examples Thanks, Alejandro
          Hide
          kamrul Mohammad Kamrul Islam added a comment -

          This issue is being addressed into Jira OOZIE-552.
          In addition, it supports multiple version.

          Show
          kamrul Mohammad Kamrul Islam added a comment - This issue is being addressed into Jira OOZIE-552 . In addition, it supports multiple version.
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          Forgot to mention the different between this JIRA & OOZIE-552.

          OOZIE-552 allows a user to specify multiple directories in the sharelib path (a la classpath), thus aggregating all JARs from those directories.

          This JIRA selects the directory to add to the classpath based on the action type. When running streaming action only streaming JARs are added, when running pig action only pig JARs are added.

          This JIRA only affects Oozie system sharelib, the functionality introduced by OOZIE-552 is still available. Furthermore they are complimentary.

          Note that for the case of conflicting dependencies between Pig and Hive OOZIE-552 does not provide a solution as the the multiple directories paths are aggregated to all actions

          Show
          tucu00 Alejandro Abdelnur added a comment - Forgot to mention the different between this JIRA & OOZIE-552 . OOZIE-552 allows a user to specify multiple directories in the sharelib path (a la classpath), thus aggregating all JARs from those directories. This JIRA selects the directory to add to the classpath based on the action type. When running streaming action only streaming JARs are added, when running pig action only pig JARs are added. This JIRA only affects Oozie system sharelib, the functionality introduced by OOZIE-552 is still available. Furthermore they are complimentary. Note that for the case of conflicting dependencies between Pig and Hive OOZIE-552 does not provide a solution as the the multiple directories paths are aggregated to all actions
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2909/
          -----------------------------------------------------------

          (Updated 2011-11-22 21:03:19.865952)

          Review request for oozie.

          Changes
          -------

          rebased to trunk (OOZIE-565)

          Summary
          -------

          breaks the Oozie sharelib into one subdir per action type

          This addresses bug OOZIE-610.
          https://issues.apache.org/jira/browse/OOZIE-610

          Diffs (updated)


          /trunk/core/pom.xml 1205165
          /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165
          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165
          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165
          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165
          /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165
          /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
          /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
          /trunk/examples/pom.xml 1205165
          /trunk/examples/src/main/apps/custom-main/job.properties 1205165
          /trunk/examples/src/main/apps/demo/job.properties 1205165
          /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
          /trunk/examples/src/main/apps/java-main/job.properties 1205165
          /trunk/examples/src/main/apps/pig/job.properties 1205165
          /trunk/examples/src/main/apps/streaming/job.properties 1205165
          /trunk/pom.xml 1205165
          /trunk/sharelib/pig/pom.xml PRE-CREATION
          /trunk/sharelib/pom.xml 1205165
          /trunk/sharelib/streaming/pom.xml PRE-CREATION
          /trunk/src/main/assemblies/distro.xml 1205165
          /trunk/src/main/assemblies/examples.xml 1205165
          /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
          /trunk/src/main/assemblies/sharelib.xml 1205165

          Diff: https://reviews.apache.org/r/2909/diff

          Testing
          -------

          built, installed and run pig and streaming examples

          Thanks,

          Alejandro

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/ ----------------------------------------------------------- (Updated 2011-11-22 21:03:19.865952) Review request for oozie. Changes ------- rebased to trunk ( OOZIE-565 ) Summary ------- breaks the Oozie sharelib into one subdir per action type This addresses bug OOZIE-610 . https://issues.apache.org/jira/browse/OOZIE-610 Diffs (updated) /trunk/core/pom.xml 1205165 /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165 /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165 /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165 /trunk/examples/pom.xml 1205165 /trunk/examples/src/main/apps/custom-main/job.properties 1205165 /trunk/examples/src/main/apps/demo/job.properties 1205165 /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165 /trunk/examples/src/main/apps/java-main/job.properties 1205165 /trunk/examples/src/main/apps/pig/job.properties 1205165 /trunk/examples/src/main/apps/streaming/job.properties 1205165 /trunk/pom.xml 1205165 /trunk/sharelib/pig/pom.xml PRE-CREATION /trunk/sharelib/pom.xml 1205165 /trunk/sharelib/streaming/pom.xml PRE-CREATION /trunk/src/main/assemblies/distro.xml 1205165 /trunk/src/main/assemblies/examples.xml 1205165 /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION /trunk/src/main/assemblies/sharelib.xml 1205165 Diff: https://reviews.apache.org/r/2909/diff Testing ------- built, installed and run pig and streaming examples Thanks, Alejandro
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2909/#review3446
          -----------------------------------------------------------

          Overall, it's an important problem.

          In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.

          Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/

          {lib, conf}, $SHARELIB/pig/0.7/{lib, conf}

          , $SHARELIB/pig/0.8/

          {lib, conf}

          etc. I'm leaning towards this way to make it consistent with multi-version support.

          /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java
          <https://reviews.apache.org/r/2909/#comment7693>

          do we need it?

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
          <https://reviews.apache.org/r/2909/#comment7695>

          so the name will be defined in each action. Or will it be better in the system configuration in oozie-default(site).xml?

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java
          <https://reviews.apache.org/r/2909/#comment7697>

          Is hard-coding a good approach? Default could be "pig".

          • Mohammad

          On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2909/

          -----------------------------------------------------------

          (Updated 2011-11-22 21:03:19)

          Review request for oozie.

          Summary

          -------

          breaks the Oozie sharelib into one subdir per action type

          This addresses bug OOZIE-610.

          https://issues.apache.org/jira/browse/OOZIE-610

          Diffs

          -----

          /trunk/core/pom.xml 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165

          /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165

          /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165

          /trunk/examples/pom.xml 1205165

          /trunk/examples/src/main/apps/custom-main/job.properties 1205165

          /trunk/examples/src/main/apps/demo/job.properties 1205165

          /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165

          /trunk/examples/src/main/apps/java-main/job.properties 1205165

          /trunk/examples/src/main/apps/pig/job.properties 1205165

          /trunk/examples/src/main/apps/streaming/job.properties 1205165

          /trunk/pom.xml 1205165

          /trunk/sharelib/pig/pom.xml PRE-CREATION

          /trunk/sharelib/pom.xml 1205165

          /trunk/sharelib/streaming/pom.xml PRE-CREATION

          /trunk/src/main/assemblies/distro.xml 1205165

          /trunk/src/main/assemblies/examples.xml 1205165

          /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION

          /trunk/src/main/assemblies/sharelib.xml 1205165

          Diff: https://reviews.apache.org/r/2909/diff

          Testing

          -------

          built, installed and run pig and streaming examples

          Thanks,

          Alejandro

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/#review3446 ----------------------------------------------------------- Overall, it's an important problem. In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars. Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/ {lib, conf}, $SHARELIB/pig/0.7/{lib, conf} , $SHARELIB/pig/0.8/ {lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support. /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java < https://reviews.apache.org/r/2909/#comment7693 > do we need it? /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java < https://reviews.apache.org/r/2909/#comment7695 > so the name will be defined in each action. Or will it be better in the system configuration in oozie-default(site).xml? /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java < https://reviews.apache.org/r/2909/#comment7697 > Is hard-coding a good approach? Default could be "pig". Mohammad On 2011-11-22 21:03:19, Alejandro Abdelnur wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/ ----------------------------------------------------------- (Updated 2011-11-22 21:03:19) Review request for oozie. Summary ------- breaks the Oozie sharelib into one subdir per action type This addresses bug OOZIE-610 . https://issues.apache.org/jira/browse/OOZIE-610 Diffs ----- /trunk/core/pom.xml 1205165 /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165 /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165 /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165 /trunk/examples/pom.xml 1205165 /trunk/examples/src/main/apps/custom-main/job.properties 1205165 /trunk/examples/src/main/apps/demo/job.properties 1205165 /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165 /trunk/examples/src/main/apps/java-main/job.properties 1205165 /trunk/examples/src/main/apps/pig/job.properties 1205165 /trunk/examples/src/main/apps/streaming/job.properties 1205165 /trunk/pom.xml 1205165 /trunk/sharelib/pig/pom.xml PRE-CREATION /trunk/sharelib/pom.xml 1205165 /trunk/sharelib/streaming/pom.xml PRE-CREATION /trunk/src/main/assemblies/distro.xml 1205165 /trunk/src/main/assemblies/examples.xml 1205165 /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION /trunk/src/main/assemblies/sharelib.xml 1205165 Diff: https://reviews.apache.org/r/2909/diff Testing ------- built, installed and run pig and streaming examples Thanks, Alejandro
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          On 2011-11-22 23:26:39, Mohammad Islam wrote:

          > Overall, it's an important problem.

          >

          > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.

          >

          > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.

          >

          >

          Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected.

          Adding multi version support is a good idea. It would also require changes in the action XML to specify the version.

          Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later.

          On 2011-11-22 23:26:39, Mohammad Islam wrote:

          > /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java, line 30

          > <https://reviews.apache.org/r/2909/diff/3/?file=59834#file59834line30>

          >

          > do we need it?

          yes, because of the signature of the getShareLibPostFix() method

          On 2011-11-22 23:26:39, Mohammad Islam wrote:

          > /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java, line 209

          > <https://reviews.apache.org/r/2909/diff/3/?file=59836#file59836line209>

          >

          > so the name will be defined in each action. Or will it be better in the system configuration in oozie-default(site).xml?

          this libraries are specific to an action type, thus it belongs to the action. Making this configurable does not seem necessary at this point.

          On 2011-11-22 23:26:39, Mohammad Islam wrote:

          > /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java, line 140

          > <https://reviews.apache.org/r/2909/diff/3/?file=59837#file59837line140>

          >

          > Is hard-coding a good approach? Default could be "pig".

          i'd start simple, later we can add more intelligence/configurability if we find it necessary, as it is in a method already, the change will be transparent if we do it.

          • Alejandro

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2909/#review3446
          -----------------------------------------------------------

          On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2909/

          -----------------------------------------------------------

          (Updated 2011-11-22 21:03:19)

          Review request for oozie.

          Summary

          -------

          breaks the Oozie sharelib into one subdir per action type

          This addresses bug OOZIE-610.

          https://issues.apache.org/jira/browse/OOZIE-610

          Diffs

          -----

          /trunk/core/pom.xml 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165

          /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165

          /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165

          /trunk/examples/pom.xml 1205165

          /trunk/examples/src/main/apps/custom-main/job.properties 1205165

          /trunk/examples/src/main/apps/demo/job.properties 1205165

          /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165

          /trunk/examples/src/main/apps/java-main/job.properties 1205165

          /trunk/examples/src/main/apps/pig/job.properties 1205165

          /trunk/examples/src/main/apps/streaming/job.properties 1205165

          /trunk/pom.xml 1205165

          /trunk/sharelib/pig/pom.xml PRE-CREATION

          /trunk/sharelib/pom.xml 1205165

          /trunk/sharelib/streaming/pom.xml PRE-CREATION

          /trunk/src/main/assemblies/distro.xml 1205165

          /trunk/src/main/assemblies/examples.xml 1205165

          /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION

          /trunk/src/main/assemblies/sharelib.xml 1205165

          Diff: https://reviews.apache.org/r/2909/diff

          Testing

          -------

          built, installed and run pig and streaming examples

          Thanks,

          Alejandro

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - On 2011-11-22 23:26:39, Mohammad Islam wrote: > Overall, it's an important problem. > > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars. > > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support. > > Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected. Adding multi version support is a good idea. It would also require changes in the action XML to specify the version. Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later. On 2011-11-22 23:26:39, Mohammad Islam wrote: > /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java, line 30 > < https://reviews.apache.org/r/2909/diff/3/?file=59834#file59834line30 > > > do we need it? yes, because of the signature of the getShareLibPostFix() method On 2011-11-22 23:26:39, Mohammad Islam wrote: > /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java, line 209 > < https://reviews.apache.org/r/2909/diff/3/?file=59836#file59836line209 > > > so the name will be defined in each action. Or will it be better in the system configuration in oozie-default(site).xml? this libraries are specific to an action type, thus it belongs to the action. Making this configurable does not seem necessary at this point. On 2011-11-22 23:26:39, Mohammad Islam wrote: > /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java, line 140 > < https://reviews.apache.org/r/2909/diff/3/?file=59837#file59837line140 > > > Is hard-coding a good approach? Default could be "pig". i'd start simple, later we can add more intelligence/configurability if we find it necessary, as it is in a method already, the change will be transparent if we do it. Alejandro ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/#review3446 ----------------------------------------------------------- On 2011-11-22 21:03:19, Alejandro Abdelnur wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/ ----------------------------------------------------------- (Updated 2011-11-22 21:03:19) Review request for oozie. Summary ------- breaks the Oozie sharelib into one subdir per action type This addresses bug OOZIE-610 . https://issues.apache.org/jira/browse/OOZIE-610 Diffs ----- /trunk/core/pom.xml 1205165 /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165 /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165 /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165 /trunk/examples/pom.xml 1205165 /trunk/examples/src/main/apps/custom-main/job.properties 1205165 /trunk/examples/src/main/apps/demo/job.properties 1205165 /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165 /trunk/examples/src/main/apps/java-main/job.properties 1205165 /trunk/examples/src/main/apps/pig/job.properties 1205165 /trunk/examples/src/main/apps/streaming/job.properties 1205165 /trunk/pom.xml 1205165 /trunk/sharelib/pig/pom.xml PRE-CREATION /trunk/sharelib/pom.xml 1205165 /trunk/sharelib/streaming/pom.xml PRE-CREATION /trunk/src/main/assemblies/distro.xml 1205165 /trunk/src/main/assemblies/examples.xml 1205165 /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION /trunk/src/main/assemblies/sharelib.xml 1205165 Diff: https://reviews.apache.org/r/2909/diff Testing ------- built, installed and run pig and streaming examples Thanks, Alejandro
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          On 2011-11-22 23:26:39, Mohammad Islam wrote:

          > Overall, it's an important problem.

          >

          > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.

          >

          > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.

          >

          >

          Alejandro Abdelnur wrote:

          Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected.

          Adding multi version support is a good idea. It would also require changes in the action XML to specify the version.

          Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later.

          I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal.

          Thanks.

          • Alejandro

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2909/#review3446
          -----------------------------------------------------------

          On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2909/

          -----------------------------------------------------------

          (Updated 2011-11-22 21:03:19)

          Review request for oozie.

          Summary

          -------

          breaks the Oozie sharelib into one subdir per action type

          This addresses bug OOZIE-610.

          https://issues.apache.org/jira/browse/OOZIE-610

          Diffs

          -----

          /trunk/core/pom.xml 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165

          /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165

          /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165

          /trunk/examples/pom.xml 1205165

          /trunk/examples/src/main/apps/custom-main/job.properties 1205165

          /trunk/examples/src/main/apps/demo/job.properties 1205165

          /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165

          /trunk/examples/src/main/apps/java-main/job.properties 1205165

          /trunk/examples/src/main/apps/pig/job.properties 1205165

          /trunk/examples/src/main/apps/streaming/job.properties 1205165

          /trunk/pom.xml 1205165

          /trunk/sharelib/pig/pom.xml PRE-CREATION

          /trunk/sharelib/pom.xml 1205165

          /trunk/sharelib/streaming/pom.xml PRE-CREATION

          /trunk/src/main/assemblies/distro.xml 1205165

          /trunk/src/main/assemblies/examples.xml 1205165

          /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION

          /trunk/src/main/assemblies/sharelib.xml 1205165

          Diff: https://reviews.apache.org/r/2909/diff

          Testing

          -------

          built, installed and run pig and streaming examples

          Thanks,

          Alejandro

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - On 2011-11-22 23:26:39, Mohammad Islam wrote: > Overall, it's an important problem. > > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars. > > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support. > > Alejandro Abdelnur wrote: Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected. Adding multi version support is a good idea. It would also require changes in the action XML to specify the version. Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later. I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal. Thanks. Alejandro ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/#review3446 ----------------------------------------------------------- On 2011-11-22 21:03:19, Alejandro Abdelnur wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/ ----------------------------------------------------------- (Updated 2011-11-22 21:03:19) Review request for oozie. Summary ------- breaks the Oozie sharelib into one subdir per action type This addresses bug OOZIE-610 . https://issues.apache.org/jira/browse/OOZIE-610 Diffs ----- /trunk/core/pom.xml 1205165 /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165 /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165 /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165 /trunk/examples/pom.xml 1205165 /trunk/examples/src/main/apps/custom-main/job.properties 1205165 /trunk/examples/src/main/apps/demo/job.properties 1205165 /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165 /trunk/examples/src/main/apps/java-main/job.properties 1205165 /trunk/examples/src/main/apps/pig/job.properties 1205165 /trunk/examples/src/main/apps/streaming/job.properties 1205165 /trunk/pom.xml 1205165 /trunk/sharelib/pig/pom.xml PRE-CREATION /trunk/sharelib/pom.xml 1205165 /trunk/sharelib/streaming/pom.xml PRE-CREATION /trunk/src/main/assemblies/distro.xml 1205165 /trunk/src/main/assemblies/examples.xml 1205165 /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION /trunk/src/main/assemblies/sharelib.xml 1205165 Diff: https://reviews.apache.org/r/2909/diff Testing ------- built, installed and run pig and streaming examples Thanks, Alejandro
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2909/#review3459
          -----------------------------------------------------------

          Overall comments:
          1. Doing it incrementally will be good idea. But we need to consider the backward compatibility. For example, currently we are reading pig/.jar, in the next oozie release, we decided to support pig/lib/.jar or pig/0.8/lib/.jar or pig/stable/lib/.jar. In this case, the old setup will not work.

          2. We wanted to support multiple version too. Virag has the patch. What is the best way of getting the benefits of both?

          /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java
          <https://reviews.apache.org/r/2909/#comment7713>

          But I don't see any other code changes in this file.

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
          <https://reviews.apache.org/r/2909/#comment7716>

          Putting this condition earlier makes more sense.

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
          <https://reviews.apache.org/r/2909/#comment7715>

          it's ok

          /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java
          <https://reviews.apache.org/r/2909/#comment7714>

          why are we not reading the system library path for jars. There could be some system jars ,that are not any product specific (like pig, hive) but are commonly used. Also the question of backward compatibility is there.

          /trunk/docs/src/site/twiki/DG_Examples.twiki
          <https://reviews.apache.org/r/2909/#comment7712>

          So user level libpath will not be supported?

          • Mohammad

          On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2909/

          -----------------------------------------------------------

          (Updated 2011-11-22 21:03:19)

          Review request for oozie.

          Summary

          -------

          breaks the Oozie sharelib into one subdir per action type

          This addresses bug OOZIE-610.

          https://issues.apache.org/jira/browse/OOZIE-610

          Diffs

          -----

          /trunk/core/pom.xml 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165

          /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165

          /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165

          /trunk/examples/pom.xml 1205165

          /trunk/examples/src/main/apps/custom-main/job.properties 1205165

          /trunk/examples/src/main/apps/demo/job.properties 1205165

          /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165

          /trunk/examples/src/main/apps/java-main/job.properties 1205165

          /trunk/examples/src/main/apps/pig/job.properties 1205165

          /trunk/examples/src/main/apps/streaming/job.properties 1205165

          /trunk/pom.xml 1205165

          /trunk/sharelib/pig/pom.xml PRE-CREATION

          /trunk/sharelib/pom.xml 1205165

          /trunk/sharelib/streaming/pom.xml PRE-CREATION

          /trunk/src/main/assemblies/distro.xml 1205165

          /trunk/src/main/assemblies/examples.xml 1205165

          /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION

          /trunk/src/main/assemblies/sharelib.xml 1205165

          Diff: https://reviews.apache.org/r/2909/diff

          Testing

          -------

          built, installed and run pig and streaming examples

          Thanks,

          Alejandro

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/#review3459 ----------------------------------------------------------- Overall comments: 1. Doing it incrementally will be good idea. But we need to consider the backward compatibility. For example, currently we are reading pig/ .jar, in the next oozie release, we decided to support pig/lib/ .jar or pig/0.8/lib/ .jar or pig/stable/lib/ .jar. In this case, the old setup will not work. 2. We wanted to support multiple version too. Virag has the patch. What is the best way of getting the benefits of both? /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java < https://reviews.apache.org/r/2909/#comment7713 > But I don't see any other code changes in this file. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java < https://reviews.apache.org/r/2909/#comment7716 > Putting this condition earlier makes more sense. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java < https://reviews.apache.org/r/2909/#comment7715 > it's ok /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java < https://reviews.apache.org/r/2909/#comment7714 > why are we not reading the system library path for jars. There could be some system jars ,that are not any product specific (like pig, hive) but are commonly used. Also the question of backward compatibility is there. /trunk/docs/src/site/twiki/DG_Examples.twiki < https://reviews.apache.org/r/2909/#comment7712 > So user level libpath will not be supported? Mohammad On 2011-11-22 21:03:19, Alejandro Abdelnur wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/ ----------------------------------------------------------- (Updated 2011-11-22 21:03:19) Review request for oozie. Summary ------- breaks the Oozie sharelib into one subdir per action type This addresses bug OOZIE-610 . https://issues.apache.org/jira/browse/OOZIE-610 Diffs ----- /trunk/core/pom.xml 1205165 /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165 /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165 /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165 /trunk/examples/pom.xml 1205165 /trunk/examples/src/main/apps/custom-main/job.properties 1205165 /trunk/examples/src/main/apps/demo/job.properties 1205165 /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165 /trunk/examples/src/main/apps/java-main/job.properties 1205165 /trunk/examples/src/main/apps/pig/job.properties 1205165 /trunk/examples/src/main/apps/streaming/job.properties 1205165 /trunk/pom.xml 1205165 /trunk/sharelib/pig/pom.xml PRE-CREATION /trunk/sharelib/pom.xml 1205165 /trunk/sharelib/streaming/pom.xml PRE-CREATION /trunk/src/main/assemblies/distro.xml 1205165 /trunk/src/main/assemblies/examples.xml 1205165 /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION /trunk/src/main/assemblies/sharelib.xml 1205165 Diff: https://reviews.apache.org/r/2909/diff Testing ------- built, installed and run pig and streaming examples Thanks, Alejandro
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          On 2011-11-23 03:31:31, Mohammad Islam wrote:

          > Overall comments:

          > 1. Doing it incrementally will be good idea. But we need to consider the backward compatibility. For example, currently we are reading pig/.jar, in the next oozie release, we decided to support pig/lib/.jar or pig/0.8/lib/.jar or pig/stable/lib/.jar. In this case, the old setup will not work.

          >

          > 2. We wanted to support multiple version too. Virag has the patch. What is the best way of getting the benefits of both?

          >

          Backwards compatibility would not be an issue because when you change the layout of the library you'll change the resolution logic, and for jobs will be transparent. When you upgrade Oozie you'll have to upgrade the sharelib at the same time; but WF apps will be unaffected. Furthermore a migration path would be that now you are only supporting one version, 'stable', and later you'll support other versions via a configuration in the action XML.

          Alternate versions of libraries could be supported by a 'oozie.action.library.version' property in the action configuration, if not set it would resolve to 'stable', if set it would resolve to the version specified.

          This change is not complex, actually it is quite simple. What complicates things a bit is the packaging of sharelibs.

          Again, we could do this incremental (as in a couple of weeks), I just don't want to delay this patch much more as I have a patch chain (this, Hive, Sqoop, Alfredo) which takes time and work to keep up to date.

          On 2011-11-23 03:31:31, Mohammad Islam wrote:

          > /trunk/docs/src/site/twiki/DG_Examples.twiki, line 95

          > <https://reviews.apache.org/r/2909/diff/3/?file=59839#file59839line95>

          >

          > So user level libpath will not be supported?

          This is still supported, the examples are not using it anymore, instead they use the Oozie share lib.

          On 2011-11-23 03:31:31, Mohammad Islam wrote:

          > /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java, line 192

          > <https://reviews.apache.org/r/2909/diff/3/?file=59838#file59838line192>

          >

          > why are we not reading the system library path for jars. There could be some system jars ,that are not any product specific (like pig, hive) but are commonly used. Also the question of backward compatibility is there.

          Is this a real scenario for you or hypothetical? If the latter then I would not worry as users can still use user set library paths.

          On 2011-11-23 03:31:31, Mohammad Islam wrote:

          > /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java, line 30

          > <https://reviews.apache.org/r/2909/diff/3/?file=59834#file59834line30>

          >

          > But I don't see any other code changes in this file.

          My bad, thought the prev comment was for the JavaActionExecutor class, will remove.

          • Alejandro

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2909/#review3459
          -----------------------------------------------------------

          On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2909/

          -----------------------------------------------------------

          (Updated 2011-11-22 21:03:19)

          Review request for oozie.

          Summary

          -------

          breaks the Oozie sharelib into one subdir per action type

          This addresses bug OOZIE-610.

          https://issues.apache.org/jira/browse/OOZIE-610

          Diffs

          -----

          /trunk/core/pom.xml 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165

          /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165

          /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165

          /trunk/examples/pom.xml 1205165

          /trunk/examples/src/main/apps/custom-main/job.properties 1205165

          /trunk/examples/src/main/apps/demo/job.properties 1205165

          /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165

          /trunk/examples/src/main/apps/java-main/job.properties 1205165

          /trunk/examples/src/main/apps/pig/job.properties 1205165

          /trunk/examples/src/main/apps/streaming/job.properties 1205165

          /trunk/pom.xml 1205165

          /trunk/sharelib/pig/pom.xml PRE-CREATION

          /trunk/sharelib/pom.xml 1205165

          /trunk/sharelib/streaming/pom.xml PRE-CREATION

          /trunk/src/main/assemblies/distro.xml 1205165

          /trunk/src/main/assemblies/examples.xml 1205165

          /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION

          /trunk/src/main/assemblies/sharelib.xml 1205165

          Diff: https://reviews.apache.org/r/2909/diff

          Testing

          -------

          built, installed and run pig and streaming examples

          Thanks,

          Alejandro

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - On 2011-11-23 03:31:31, Mohammad Islam wrote: > Overall comments: > 1. Doing it incrementally will be good idea. But we need to consider the backward compatibility. For example, currently we are reading pig/ .jar, in the next oozie release, we decided to support pig/lib/ .jar or pig/0.8/lib/ .jar or pig/stable/lib/ .jar. In this case, the old setup will not work. > > 2. We wanted to support multiple version too. Virag has the patch. What is the best way of getting the benefits of both? > Backwards compatibility would not be an issue because when you change the layout of the library you'll change the resolution logic, and for jobs will be transparent. When you upgrade Oozie you'll have to upgrade the sharelib at the same time; but WF apps will be unaffected. Furthermore a migration path would be that now you are only supporting one version, 'stable', and later you'll support other versions via a configuration in the action XML. Alternate versions of libraries could be supported by a 'oozie.action.library.version' property in the action configuration, if not set it would resolve to 'stable', if set it would resolve to the version specified. This change is not complex, actually it is quite simple. What complicates things a bit is the packaging of sharelibs. Again, we could do this incremental (as in a couple of weeks), I just don't want to delay this patch much more as I have a patch chain (this, Hive, Sqoop, Alfredo) which takes time and work to keep up to date. On 2011-11-23 03:31:31, Mohammad Islam wrote: > /trunk/docs/src/site/twiki/DG_Examples.twiki, line 95 > < https://reviews.apache.org/r/2909/diff/3/?file=59839#file59839line95 > > > So user level libpath will not be supported? This is still supported, the examples are not using it anymore, instead they use the Oozie share lib. On 2011-11-23 03:31:31, Mohammad Islam wrote: > /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java, line 192 > < https://reviews.apache.org/r/2909/diff/3/?file=59838#file59838line192 > > > why are we not reading the system library path for jars. There could be some system jars ,that are not any product specific (like pig, hive) but are commonly used. Also the question of backward compatibility is there. Is this a real scenario for you or hypothetical? If the latter then I would not worry as users can still use user set library paths. On 2011-11-23 03:31:31, Mohammad Islam wrote: > /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java, line 30 > < https://reviews.apache.org/r/2909/diff/3/?file=59834#file59834line30 > > > But I don't see any other code changes in this file. My bad, thought the prev comment was for the JavaActionExecutor class, will remove. Alejandro ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/#review3459 ----------------------------------------------------------- On 2011-11-22 21:03:19, Alejandro Abdelnur wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/ ----------------------------------------------------------- (Updated 2011-11-22 21:03:19) Review request for oozie. Summary ------- breaks the Oozie sharelib into one subdir per action type This addresses bug OOZIE-610 . https://issues.apache.org/jira/browse/OOZIE-610 Diffs ----- /trunk/core/pom.xml 1205165 /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165 /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165 /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165 /trunk/examples/pom.xml 1205165 /trunk/examples/src/main/apps/custom-main/job.properties 1205165 /trunk/examples/src/main/apps/demo/job.properties 1205165 /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165 /trunk/examples/src/main/apps/java-main/job.properties 1205165 /trunk/examples/src/main/apps/pig/job.properties 1205165 /trunk/examples/src/main/apps/streaming/job.properties 1205165 /trunk/pom.xml 1205165 /trunk/sharelib/pig/pom.xml PRE-CREATION /trunk/sharelib/pom.xml 1205165 /trunk/sharelib/streaming/pom.xml PRE-CREATION /trunk/src/main/assemblies/distro.xml 1205165 /trunk/src/main/assemblies/examples.xml 1205165 /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION /trunk/src/main/assemblies/sharelib.xml 1205165 Diff: https://reviews.apache.org/r/2909/diff Testing ------- built, installed and run pig and streaming examples Thanks, Alejandro
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          On 2011-11-22 23:26:39, Mohammad Islam wrote:

          > Overall, it's an important problem.

          >

          > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.

          >

          > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.

          >

          >

          Alejandro Abdelnur wrote:

          Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected.

          Adding multi version support is a good idea. It would also require changes in the action XML to specify the version.

          Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later.

          Alejandro Abdelnur wrote:

          I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal.

          Thanks.

          I think the "use_version" support is considered here.
          Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2
          Please review this patch.

          We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system)
          https://issues.apache.org/jira/browse/OOZIE-555

          • Mohammad

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2909/#review3446
          -----------------------------------------------------------

          On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2909/

          -----------------------------------------------------------

          (Updated 2011-11-22 21:03:19)

          Review request for oozie.

          Summary

          -------

          breaks the Oozie sharelib into one subdir per action type

          This addresses bug OOZIE-610.

          https://issues.apache.org/jira/browse/OOZIE-610

          Diffs

          -----

          /trunk/core/pom.xml 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165

          /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165

          /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165

          /trunk/examples/pom.xml 1205165

          /trunk/examples/src/main/apps/custom-main/job.properties 1205165

          /trunk/examples/src/main/apps/demo/job.properties 1205165

          /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165

          /trunk/examples/src/main/apps/java-main/job.properties 1205165

          /trunk/examples/src/main/apps/pig/job.properties 1205165

          /trunk/examples/src/main/apps/streaming/job.properties 1205165

          /trunk/pom.xml 1205165

          /trunk/sharelib/pig/pom.xml PRE-CREATION

          /trunk/sharelib/pom.xml 1205165

          /trunk/sharelib/streaming/pom.xml PRE-CREATION

          /trunk/src/main/assemblies/distro.xml 1205165

          /trunk/src/main/assemblies/examples.xml 1205165

          /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION

          /trunk/src/main/assemblies/sharelib.xml 1205165

          Diff: https://reviews.apache.org/r/2909/diff

          Testing

          -------

          built, installed and run pig and streaming examples

          Thanks,

          Alejandro

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - On 2011-11-22 23:26:39, Mohammad Islam wrote: > Overall, it's an important problem. > > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars. > > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support. > > Alejandro Abdelnur wrote: Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected. Adding multi version support is a good idea. It would also require changes in the action XML to specify the version. Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later. Alejandro Abdelnur wrote: I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal. Thanks. I think the "use_version" support is considered here. Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2 Please review this patch. We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system) https://issues.apache.org/jira/browse/OOZIE-555 Mohammad ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/#review3446 ----------------------------------------------------------- On 2011-11-22 21:03:19, Alejandro Abdelnur wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/ ----------------------------------------------------------- (Updated 2011-11-22 21:03:19) Review request for oozie. Summary ------- breaks the Oozie sharelib into one subdir per action type This addresses bug OOZIE-610 . https://issues.apache.org/jira/browse/OOZIE-610 Diffs ----- /trunk/core/pom.xml 1205165 /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165 /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165 /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165 /trunk/examples/pom.xml 1205165 /trunk/examples/src/main/apps/custom-main/job.properties 1205165 /trunk/examples/src/main/apps/demo/job.properties 1205165 /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165 /trunk/examples/src/main/apps/java-main/job.properties 1205165 /trunk/examples/src/main/apps/pig/job.properties 1205165 /trunk/examples/src/main/apps/streaming/job.properties 1205165 /trunk/pom.xml 1205165 /trunk/sharelib/pig/pom.xml PRE-CREATION /trunk/sharelib/pom.xml 1205165 /trunk/sharelib/streaming/pom.xml PRE-CREATION /trunk/src/main/assemblies/distro.xml 1205165 /trunk/src/main/assemblies/examples.xml 1205165 /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION /trunk/src/main/assemblies/sharelib.xml 1205165 Diff: https://reviews.apache.org/r/2909/diff Testing ------- built, installed and run pig and streaming examples Thanks, Alejandro
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          On 2011-11-22 23:26:39, Mohammad Islam wrote:

          > Overall, it's an important problem.

          >

          > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.

          >

          > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.

          >

          >

          Alejandro Abdelnur wrote:

          Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected.

          Adding multi version support is a good idea. It would also require changes in the action XML to specify the version.

          Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later.

          Alejandro Abdelnur wrote:

          I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal.

          Thanks.

          Mohammad Islam wrote:

          I think the "use_version" support is considered here.

          Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2

          Please review this patch.

          We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system)

          https://issues.apache.org/jira/browse/OOZIE-555

          Mmmh (regarding my lost comments).

          OK, the idea is similar, the main difference with what I'm proposing is that in Virag's patch it requires a new WF schema and that the library paths are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes that make difficult to see what is a change and what is not)

          To achieve similar functionality on top of OOZIE-610 we could:

          • Change the sharelib path to contain action library versions, ie:

          share/lib/streaming/0.20.204.0/
          share/lib/pig/0.8.0/
          share/lib/pig/0.9.0/

          • Have an oozie-site configuration, per action that defines the stable version of the library, ie:

          oozie.action.stable.library.version.streaming=0.20.204.0
          oozie.action.stable.library.version.pig=0.8.0

          • The ActionExecutor getShareLibPostFix() method would have the following logic:

          getShareLibPostFix(...) {

          String version = action.config.get("oozie.action.library.version");
          if (version == null)

          { version = oozie.config.get("oozie.action.stable.library.version"); }

          if (version != null)

          { //construct the library path using the specified version }

          }

          This approach does not require changes in the WF schema, instead uses a configuration value in case you want a different version than the stable, if not it just works for existing WF apps.

          If we are OK with this approach I'll file a JIRA with it.

          • Alejandro

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2909/#review3446
          -----------------------------------------------------------

          On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2909/

          -----------------------------------------------------------

          (Updated 2011-11-22 21:03:19)

          Review request for oozie.

          Summary

          -------

          breaks the Oozie sharelib into one subdir per action type

          This addresses bug OOZIE-610.

          https://issues.apache.org/jira/browse/OOZIE-610

          Diffs

          -----

          /trunk/core/pom.xml 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165

          /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165

          /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165

          /trunk/examples/pom.xml 1205165

          /trunk/examples/src/main/apps/custom-main/job.properties 1205165

          /trunk/examples/src/main/apps/demo/job.properties 1205165

          /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165

          /trunk/examples/src/main/apps/java-main/job.properties 1205165

          /trunk/examples/src/main/apps/pig/job.properties 1205165

          /trunk/examples/src/main/apps/streaming/job.properties 1205165

          /trunk/pom.xml 1205165

          /trunk/sharelib/pig/pom.xml PRE-CREATION

          /trunk/sharelib/pom.xml 1205165

          /trunk/sharelib/streaming/pom.xml PRE-CREATION

          /trunk/src/main/assemblies/distro.xml 1205165

          /trunk/src/main/assemblies/examples.xml 1205165

          /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION

          /trunk/src/main/assemblies/sharelib.xml 1205165

          Diff: https://reviews.apache.org/r/2909/diff

          Testing

          -------

          built, installed and run pig and streaming examples

          Thanks,

          Alejandro

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - On 2011-11-22 23:26:39, Mohammad Islam wrote: > Overall, it's an important problem. > > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars. > > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support. > > Alejandro Abdelnur wrote: Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected. Adding multi version support is a good idea. It would also require changes in the action XML to specify the version. Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later. Alejandro Abdelnur wrote: I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal. Thanks. Mohammad Islam wrote: I think the "use_version" support is considered here. Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2 Please review this patch. We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system) https://issues.apache.org/jira/browse/OOZIE-555 Mmmh (regarding my lost comments). OK, the idea is similar, the main difference with what I'm proposing is that in Virag's patch it requires a new WF schema and that the library paths are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes that make difficult to see what is a change and what is not) To achieve similar functionality on top of OOZIE-610 we could: Change the sharelib path to contain action library versions, ie: share/lib/streaming/0.20.204.0/ share/lib/pig/0.8.0/ share/lib/pig/0.9.0/ Have an oozie-site configuration, per action that defines the stable version of the library, ie: oozie.action.stable.library.version.streaming=0.20.204.0 oozie.action.stable.library.version.pig=0.8.0 The ActionExecutor getShareLibPostFix() method would have the following logic: getShareLibPostFix(...) { String version = action.config.get("oozie.action.library.version"); if (version == null) { version = oozie.config.get("oozie.action.stable.library.version"); } if (version != null) { //construct the library path using the specified version } } This approach does not require changes in the WF schema, instead uses a configuration value in case you want a different version than the stable, if not it just works for existing WF apps. If we are OK with this approach I'll file a JIRA with it. Alejandro ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/#review3446 ----------------------------------------------------------- On 2011-11-22 21:03:19, Alejandro Abdelnur wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/ ----------------------------------------------------------- (Updated 2011-11-22 21:03:19) Review request for oozie. Summary ------- breaks the Oozie sharelib into one subdir per action type This addresses bug OOZIE-610 . https://issues.apache.org/jira/browse/OOZIE-610 Diffs ----- /trunk/core/pom.xml 1205165 /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165 /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165 /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165 /trunk/examples/pom.xml 1205165 /trunk/examples/src/main/apps/custom-main/job.properties 1205165 /trunk/examples/src/main/apps/demo/job.properties 1205165 /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165 /trunk/examples/src/main/apps/java-main/job.properties 1205165 /trunk/examples/src/main/apps/pig/job.properties 1205165 /trunk/examples/src/main/apps/streaming/job.properties 1205165 /trunk/pom.xml 1205165 /trunk/sharelib/pig/pom.xml PRE-CREATION /trunk/sharelib/pom.xml 1205165 /trunk/sharelib/streaming/pom.xml PRE-CREATION /trunk/src/main/assemblies/distro.xml 1205165 /trunk/src/main/assemblies/examples.xml 1205165 /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION /trunk/src/main/assemblies/sharelib.xml 1205165 Diff: https://reviews.apache.org/r/2909/diff Testing ------- built, installed and run pig and streaming examples Thanks, Alejandro
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          On 2011-11-22 23:26:39, Mohammad Islam wrote:

          > Overall, it's an important problem.

          >

          > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.

          >

          > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.

          >

          >

          Alejandro Abdelnur wrote:

          Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected.

          Adding multi version support is a good idea. It would also require changes in the action XML to specify the version.

          Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later.

          Alejandro Abdelnur wrote:

          I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal.

          Thanks.

          Mohammad Islam wrote:

          I think the "use_version" support is considered here.

          Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2

          Please review this patch.

          We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system)

          https://issues.apache.org/jira/browse/OOZIE-555

          Alejandro Abdelnur wrote:

          Mmmh (regarding my lost comments).

          OK, the idea is similar, the main difference with what I'm proposing is that in Virag's patch it requires a new WF schema and that the library paths are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes that make difficult to see what is a change and what is not)

          To achieve similar functionality on top of OOZIE-610 we could:

          * Change the sharelib path to contain action library versions, ie:

          share/lib/streaming/0.20.204.0/

          share/lib/pig/0.8.0/

          share/lib/pig/0.9.0/

          * Have an oozie-site configuration, per action that defines the stable version of the library, ie:

          oozie.action.stable.library.version.streaming=0.20.204.0

          oozie.action.stable.library.version.pig=0.8.0

          * The ActionExecutor getShareLibPostFix() method would have the following logic:

          getShareLibPostFix(...) {

          String version = action.config.get("oozie.action.library.version");

          if (version == null) { bq. version = oozie.config.get("oozie.action.stable.library.version"); bq. }

          if (version != null) { bq. //construct the library path using the specified version bq. }

          }

          This approach does not require changes in the WF schema, instead uses a configuration value in case you want a different version than the stable, if not it just works for existing WF apps.

          If we are OK with this approach I'll file a JIRA with it.

          In addition to your proposal, we want to give the user the flexibility to choose the version optionally. User can specify that in wf.xml using use-version. Basically the effective version will be user-specified version , if it is defined in wf.xml. Otherwise, the default one that you mentioned will be the effective version.

          there is already a JIRA on this:
          https://issues.apache.org/jira/browse/OOZIE-555
          You can add your comments there.

          • Mohammad

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2909/#review3446
          -----------------------------------------------------------

          On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2909/

          -----------------------------------------------------------

          (Updated 2011-11-22 21:03:19)

          Review request for oozie.

          Summary

          -------

          breaks the Oozie sharelib into one subdir per action type

          This addresses bug OOZIE-610.

          https://issues.apache.org/jira/browse/OOZIE-610

          Diffs

          -----

          /trunk/core/pom.xml 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165

          /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165

          /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165

          /trunk/examples/pom.xml 1205165

          /trunk/examples/src/main/apps/custom-main/job.properties 1205165

          /trunk/examples/src/main/apps/demo/job.properties 1205165

          /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165

          /trunk/examples/src/main/apps/java-main/job.properties 1205165

          /trunk/examples/src/main/apps/pig/job.properties 1205165

          /trunk/examples/src/main/apps/streaming/job.properties 1205165

          /trunk/pom.xml 1205165

          /trunk/sharelib/pig/pom.xml PRE-CREATION

          /trunk/sharelib/pom.xml 1205165

          /trunk/sharelib/streaming/pom.xml PRE-CREATION

          /trunk/src/main/assemblies/distro.xml 1205165

          /trunk/src/main/assemblies/examples.xml 1205165

          /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION

          /trunk/src/main/assemblies/sharelib.xml 1205165

          Diff: https://reviews.apache.org/r/2909/diff

          Testing

          -------

          built, installed and run pig and streaming examples

          Thanks,

          Alejandro

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - On 2011-11-22 23:26:39, Mohammad Islam wrote: > Overall, it's an important problem. > > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars. > > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support. > > Alejandro Abdelnur wrote: Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected. Adding multi version support is a good idea. It would also require changes in the action XML to specify the version. Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later. Alejandro Abdelnur wrote: I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal. Thanks. Mohammad Islam wrote: I think the "use_version" support is considered here. Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2 Please review this patch. We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system) https://issues.apache.org/jira/browse/OOZIE-555 Alejandro Abdelnur wrote: Mmmh (regarding my lost comments). OK, the idea is similar, the main difference with what I'm proposing is that in Virag's patch it requires a new WF schema and that the library paths are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes that make difficult to see what is a change and what is not) To achieve similar functionality on top of OOZIE-610 we could: * Change the sharelib path to contain action library versions, ie: share/lib/streaming/0.20.204.0/ share/lib/pig/0.8.0/ share/lib/pig/0.9.0/ * Have an oozie-site configuration, per action that defines the stable version of the library, ie: oozie.action.stable.library.version.streaming=0.20.204.0 oozie.action.stable.library.version.pig=0.8.0 * The ActionExecutor getShareLibPostFix() method would have the following logic: getShareLibPostFix(...) { String version = action.config.get("oozie.action.library.version"); if (version == null) { bq. version = oozie.config.get("oozie.action.stable.library.version"); bq. } if (version != null) { bq. //construct the library path using the specified version bq. } } This approach does not require changes in the WF schema, instead uses a configuration value in case you want a different version than the stable, if not it just works for existing WF apps. If we are OK with this approach I'll file a JIRA with it. In addition to your proposal, we want to give the user the flexibility to choose the version optionally . User can specify that in wf.xml using use-version. Basically the effective version will be user-specified version , if it is defined in wf.xml. Otherwise, the default one that you mentioned will be the effective version. there is already a JIRA on this: https://issues.apache.org/jira/browse/OOZIE-555 You can add your comments there. Mohammad ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/#review3446 ----------------------------------------------------------- On 2011-11-22 21:03:19, Alejandro Abdelnur wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/ ----------------------------------------------------------- (Updated 2011-11-22 21:03:19) Review request for oozie. Summary ------- breaks the Oozie sharelib into one subdir per action type This addresses bug OOZIE-610 . https://issues.apache.org/jira/browse/OOZIE-610 Diffs ----- /trunk/core/pom.xml 1205165 /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165 /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165 /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165 /trunk/examples/pom.xml 1205165 /trunk/examples/src/main/apps/custom-main/job.properties 1205165 /trunk/examples/src/main/apps/demo/job.properties 1205165 /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165 /trunk/examples/src/main/apps/java-main/job.properties 1205165 /trunk/examples/src/main/apps/pig/job.properties 1205165 /trunk/examples/src/main/apps/streaming/job.properties 1205165 /trunk/pom.xml 1205165 /trunk/sharelib/pig/pom.xml PRE-CREATION /trunk/sharelib/pom.xml 1205165 /trunk/sharelib/streaming/pom.xml PRE-CREATION /trunk/src/main/assemblies/distro.xml 1205165 /trunk/src/main/assemblies/examples.xml 1205165 /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION /trunk/src/main/assemblies/sharelib.xml 1205165 Diff: https://reviews.apache.org/r/2909/diff Testing ------- built, installed and run pig and streaming examples Thanks, Alejandro
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          On 2011-11-22 23:26:39, Mohammad Islam wrote:

          > Overall, it's an important problem.

          >

          > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.

          >

          > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.

          >

          >

          Alejandro Abdelnur wrote:

          Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected.

          Adding multi version support is a good idea. It would also require changes in the action XML to specify the version.

          Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later.

          Alejandro Abdelnur wrote:

          I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal.

          Thanks.

          Mohammad Islam wrote:

          I think the "use_version" support is considered here.

          Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2

          Please review this patch.

          We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system)

          https://issues.apache.org/jira/browse/OOZIE-555

          Alejandro Abdelnur wrote:

          Mmmh (regarding my lost comments).

          OK, the idea is similar, the main difference with what I'm proposing is that in Virag's patch it requires a new WF schema and that the library paths are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes that make difficult to see what is a change and what is not)

          To achieve similar functionality on top of OOZIE-610 we could:

          * Change the sharelib path to contain action library versions, ie:

          share/lib/streaming/0.20.204.0/

          share/lib/pig/0.8.0/

          share/lib/pig/0.9.0/

          * Have an oozie-site configuration, per action that defines the stable version of the library, ie:

          oozie.action.stable.library.version.streaming=0.20.204.0

          oozie.action.stable.library.version.pig=0.8.0

          * The ActionExecutor getShareLibPostFix() method would have the following logic:

          getShareLibPostFix(...) {

          String version = action.config.get("oozie.action.library.version");

          if (version == null) { bq. version = oozie.config.get("oozie.action.stable.library.version"); bq. }

          if (version != null) { bq. //construct the library path using the specified version bq. }

          }

          This approach does not require changes in the WF schema, instead uses a configuration value in case you want a different version than the stable, if not it just works for existing WF apps.

          If we are OK with this approach I'll file a JIRA with it.

          Mohammad Islam wrote:

          In addition to your proposal, we want to give the user the flexibility to choose the version optionally. User can specify that in wf.xml using use-version. Basically the effective version will be user-specified version , if it is defined in wf.xml. Otherwise, the default one that you mentioned will be the effective version.

          there is already a JIRA on this:

          https://issues.apache.org/jira/browse/OOZIE-555

          You can add your comments there.

          In my proposal, if the user sets the configuration property 'oozie.action.library.version' to any value, that version will be used. If the user does not set it the default stable one will be used.

          I'll put my proposal in OOZIE-555

          • Alejandro

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2909/#review3446
          -----------------------------------------------------------

          On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2909/

          -----------------------------------------------------------

          (Updated 2011-11-22 21:03:19)

          Review request for oozie.

          Summary

          -------

          breaks the Oozie sharelib into one subdir per action type

          This addresses bug OOZIE-610.

          https://issues.apache.org/jira/browse/OOZIE-610

          Diffs

          -----

          /trunk/core/pom.xml 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165

          /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165

          /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165

          /trunk/examples/pom.xml 1205165

          /trunk/examples/src/main/apps/custom-main/job.properties 1205165

          /trunk/examples/src/main/apps/demo/job.properties 1205165

          /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165

          /trunk/examples/src/main/apps/java-main/job.properties 1205165

          /trunk/examples/src/main/apps/pig/job.properties 1205165

          /trunk/examples/src/main/apps/streaming/job.properties 1205165

          /trunk/pom.xml 1205165

          /trunk/sharelib/pig/pom.xml PRE-CREATION

          /trunk/sharelib/pom.xml 1205165

          /trunk/sharelib/streaming/pom.xml PRE-CREATION

          /trunk/src/main/assemblies/distro.xml 1205165

          /trunk/src/main/assemblies/examples.xml 1205165

          /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION

          /trunk/src/main/assemblies/sharelib.xml 1205165

          Diff: https://reviews.apache.org/r/2909/diff

          Testing

          -------

          built, installed and run pig and streaming examples

          Thanks,

          Alejandro

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - On 2011-11-22 23:26:39, Mohammad Islam wrote: > Overall, it's an important problem. > > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars. > > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support. > > Alejandro Abdelnur wrote: Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected. Adding multi version support is a good idea. It would also require changes in the action XML to specify the version. Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later. Alejandro Abdelnur wrote: I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal. Thanks. Mohammad Islam wrote: I think the "use_version" support is considered here. Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2 Please review this patch. We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system) https://issues.apache.org/jira/browse/OOZIE-555 Alejandro Abdelnur wrote: Mmmh (regarding my lost comments). OK, the idea is similar, the main difference with what I'm proposing is that in Virag's patch it requires a new WF schema and that the library paths are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes that make difficult to see what is a change and what is not) To achieve similar functionality on top of OOZIE-610 we could: * Change the sharelib path to contain action library versions, ie: share/lib/streaming/0.20.204.0/ share/lib/pig/0.8.0/ share/lib/pig/0.9.0/ * Have an oozie-site configuration, per action that defines the stable version of the library, ie: oozie.action.stable.library.version.streaming=0.20.204.0 oozie.action.stable.library.version.pig=0.8.0 * The ActionExecutor getShareLibPostFix() method would have the following logic: getShareLibPostFix(...) { String version = action.config.get("oozie.action.library.version"); if (version == null) { bq. version = oozie.config.get("oozie.action.stable.library.version"); bq. } if (version != null) { bq. //construct the library path using the specified version bq. } } This approach does not require changes in the WF schema, instead uses a configuration value in case you want a different version than the stable, if not it just works for existing WF apps. If we are OK with this approach I'll file a JIRA with it. Mohammad Islam wrote: In addition to your proposal, we want to give the user the flexibility to choose the version optionally . User can specify that in wf.xml using use-version. Basically the effective version will be user-specified version , if it is defined in wf.xml. Otherwise, the default one that you mentioned will be the effective version. there is already a JIRA on this: https://issues.apache.org/jira/browse/OOZIE-555 You can add your comments there. In my proposal, if the user sets the configuration property 'oozie.action.library.version' to any value, that version will be used. If the user does not set it the default stable one will be used. I'll put my proposal in OOZIE-555 Alejandro ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/#review3446 ----------------------------------------------------------- On 2011-11-22 21:03:19, Alejandro Abdelnur wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/ ----------------------------------------------------------- (Updated 2011-11-22 21:03:19) Review request for oozie. Summary ------- breaks the Oozie sharelib into one subdir per action type This addresses bug OOZIE-610 . https://issues.apache.org/jira/browse/OOZIE-610 Diffs ----- /trunk/core/pom.xml 1205165 /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165 /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165 /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165 /trunk/examples/pom.xml 1205165 /trunk/examples/src/main/apps/custom-main/job.properties 1205165 /trunk/examples/src/main/apps/demo/job.properties 1205165 /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165 /trunk/examples/src/main/apps/java-main/job.properties 1205165 /trunk/examples/src/main/apps/pig/job.properties 1205165 /trunk/examples/src/main/apps/streaming/job.properties 1205165 /trunk/pom.xml 1205165 /trunk/sharelib/pig/pom.xml PRE-CREATION /trunk/sharelib/pom.xml 1205165 /trunk/sharelib/streaming/pom.xml PRE-CREATION /trunk/src/main/assemblies/distro.xml 1205165 /trunk/src/main/assemblies/examples.xml 1205165 /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION /trunk/src/main/assemblies/sharelib.xml 1205165 Diff: https://reviews.apache.org/r/2909/diff Testing ------- built, installed and run pig and streaming examples Thanks, Alejandro
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          On 2011-11-22 23:26:39, Mohammad Islam wrote:

          > Overall, it's an important problem.

          >

          > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.

          >

          > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.

          >

          >

          Alejandro Abdelnur wrote:

          Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected.

          Adding multi version support is a good idea. It would also require changes in the action XML to specify the version.

          Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later.

          Alejandro Abdelnur wrote:

          I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal.

          Thanks.

          Mohammad Islam wrote:

          I think the "use_version" support is considered here.

          Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2

          Please review this patch.

          We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system)

          https://issues.apache.org/jira/browse/OOZIE-555

          Alejandro Abdelnur wrote:

          Mmmh (regarding my lost comments).

          OK, the idea is similar, the main difference with what I'm proposing is that in Virag's patch it requires a new WF schema and that the library paths are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes that make difficult to see what is a change and what is not)

          To achieve similar functionality on top of OOZIE-610 we could:

          * Change the sharelib path to contain action library versions, ie:

          share/lib/streaming/0.20.204.0/

          share/lib/pig/0.8.0/

          share/lib/pig/0.9.0/

          * Have an oozie-site configuration, per action that defines the stable version of the library, ie:

          oozie.action.stable.library.version.streaming=0.20.204.0

          oozie.action.stable.library.version.pig=0.8.0

          * The ActionExecutor getShareLibPostFix() method would have the following logic:

          getShareLibPostFix(...) {

          String version = action.config.get("oozie.action.library.version");

          if (version == null) { bq. version = oozie.config.get("oozie.action.stable.library.version"); bq. }

          if (version != null) { bq. //construct the library path using the specified version bq. }

          }

          This approach does not require changes in the WF schema, instead uses a configuration value in case you want a different version than the stable, if not it just works for existing WF apps.

          If we are OK with this approach I'll file a JIRA with it.

          Mohammad Islam wrote:

          In addition to your proposal, we want to give the user the flexibility to choose the version optionally. User can specify that in wf.xml using use-version. Basically the effective version will be user-specified version , if it is defined in wf.xml. Otherwise, the default one that you mentioned will be the effective version.

          there is already a JIRA on this:

          https://issues.apache.org/jira/browse/OOZIE-555

          You can add your comments there.

          Alejandro Abdelnur wrote:

          In my proposal, if the user sets the configuration property 'oozie.action.library.version' to any value, that version will be used. If the user does not set it the default stable one will be used.

          I'll put my proposal in OOZIE-555

          The end-user who writes the workflow.xml should have a way of controlling his version. For example, one user tested his pig script againist pig 0.8 another one has used the latest (0.9). We should allow the both users to support that provide pig 0.8 and pig 0.9 are supported versions. I think it should be defined per-action level.

          Also we had the same design discussion few months back.

          • Mohammad

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2909/#review3446
          -----------------------------------------------------------

          On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2909/

          -----------------------------------------------------------

          (Updated 2011-11-22 21:03:19)

          Review request for oozie.

          Summary

          -------

          breaks the Oozie sharelib into one subdir per action type

          This addresses bug OOZIE-610.

          https://issues.apache.org/jira/browse/OOZIE-610

          Diffs

          -----

          /trunk/core/pom.xml 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165

          /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165

          /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165

          /trunk/examples/pom.xml 1205165

          /trunk/examples/src/main/apps/custom-main/job.properties 1205165

          /trunk/examples/src/main/apps/demo/job.properties 1205165

          /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165

          /trunk/examples/src/main/apps/java-main/job.properties 1205165

          /trunk/examples/src/main/apps/pig/job.properties 1205165

          /trunk/examples/src/main/apps/streaming/job.properties 1205165

          /trunk/pom.xml 1205165

          /trunk/sharelib/pig/pom.xml PRE-CREATION

          /trunk/sharelib/pom.xml 1205165

          /trunk/sharelib/streaming/pom.xml PRE-CREATION

          /trunk/src/main/assemblies/distro.xml 1205165

          /trunk/src/main/assemblies/examples.xml 1205165

          /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION

          /trunk/src/main/assemblies/sharelib.xml 1205165

          Diff: https://reviews.apache.org/r/2909/diff

          Testing

          -------

          built, installed and run pig and streaming examples

          Thanks,

          Alejandro

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - On 2011-11-22 23:26:39, Mohammad Islam wrote: > Overall, it's an important problem. > > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars. > > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support. > > Alejandro Abdelnur wrote: Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected. Adding multi version support is a good idea. It would also require changes in the action XML to specify the version. Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later. Alejandro Abdelnur wrote: I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal. Thanks. Mohammad Islam wrote: I think the "use_version" support is considered here. Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2 Please review this patch. We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system) https://issues.apache.org/jira/browse/OOZIE-555 Alejandro Abdelnur wrote: Mmmh (regarding my lost comments). OK, the idea is similar, the main difference with what I'm proposing is that in Virag's patch it requires a new WF schema and that the library paths are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes that make difficult to see what is a change and what is not) To achieve similar functionality on top of OOZIE-610 we could: * Change the sharelib path to contain action library versions, ie: share/lib/streaming/0.20.204.0/ share/lib/pig/0.8.0/ share/lib/pig/0.9.0/ * Have an oozie-site configuration, per action that defines the stable version of the library, ie: oozie.action.stable.library.version.streaming=0.20.204.0 oozie.action.stable.library.version.pig=0.8.0 * The ActionExecutor getShareLibPostFix() method would have the following logic: getShareLibPostFix(...) { String version = action.config.get("oozie.action.library.version"); if (version == null) { bq. version = oozie.config.get("oozie.action.stable.library.version"); bq. } if (version != null) { bq. //construct the library path using the specified version bq. } } This approach does not require changes in the WF schema, instead uses a configuration value in case you want a different version than the stable, if not it just works for existing WF apps. If we are OK with this approach I'll file a JIRA with it. Mohammad Islam wrote: In addition to your proposal, we want to give the user the flexibility to choose the version optionally . User can specify that in wf.xml using use-version. Basically the effective version will be user-specified version , if it is defined in wf.xml. Otherwise, the default one that you mentioned will be the effective version. there is already a JIRA on this: https://issues.apache.org/jira/browse/OOZIE-555 You can add your comments there. Alejandro Abdelnur wrote: In my proposal, if the user sets the configuration property 'oozie.action.library.version' to any value, that version will be used. If the user does not set it the default stable one will be used. I'll put my proposal in OOZIE-555 The end-user who writes the workflow.xml should have a way of controlling his version. For example, one user tested his pig script againist pig 0.8 another one has used the latest (0.9). We should allow the both users to support that provide pig 0.8 and pig 0.9 are supported versions. I think it should be defined per-action level. Also we had the same design discussion few months back. Mohammad ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/#review3446 ----------------------------------------------------------- On 2011-11-22 21:03:19, Alejandro Abdelnur wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/ ----------------------------------------------------------- (Updated 2011-11-22 21:03:19) Review request for oozie. Summary ------- breaks the Oozie sharelib into one subdir per action type This addresses bug OOZIE-610 . https://issues.apache.org/jira/browse/OOZIE-610 Diffs ----- /trunk/core/pom.xml 1205165 /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165 /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165 /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165 /trunk/examples/pom.xml 1205165 /trunk/examples/src/main/apps/custom-main/job.properties 1205165 /trunk/examples/src/main/apps/demo/job.properties 1205165 /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165 /trunk/examples/src/main/apps/java-main/job.properties 1205165 /trunk/examples/src/main/apps/pig/job.properties 1205165 /trunk/examples/src/main/apps/streaming/job.properties 1205165 /trunk/pom.xml 1205165 /trunk/sharelib/pig/pom.xml PRE-CREATION /trunk/sharelib/pom.xml 1205165 /trunk/sharelib/streaming/pom.xml PRE-CREATION /trunk/src/main/assemblies/distro.xml 1205165 /trunk/src/main/assemblies/examples.xml 1205165 /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION /trunk/src/main/assemblies/sharelib.xml 1205165 Diff: https://reviews.apache.org/r/2909/diff Testing ------- built, installed and run pig and streaming examples Thanks, Alejandro
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          On 2011-11-22 23:26:39, Mohammad Islam wrote:

          > Overall, it's an important problem.

          >

          > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.

          >

          > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.

          >

          >

          Alejandro Abdelnur wrote:

          Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected.

          Adding multi version support is a good idea. It would also require changes in the action XML to specify the version.

          Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later.

          Alejandro Abdelnur wrote:

          I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal.

          Thanks.

          Mohammad Islam wrote:

          I think the "use_version" support is considered here.

          Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2

          Please review this patch.

          We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system)

          https://issues.apache.org/jira/browse/OOZIE-555

          Alejandro Abdelnur wrote:

          Mmmh (regarding my lost comments).

          OK, the idea is similar, the main difference with what I'm proposing is that in Virag's patch it requires a new WF schema and that the library paths are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes that make difficult to see what is a change and what is not)

          To achieve similar functionality on top of OOZIE-610 we could:

          * Change the sharelib path to contain action library versions, ie:

          share/lib/streaming/0.20.204.0/

          share/lib/pig/0.8.0/

          share/lib/pig/0.9.0/

          * Have an oozie-site configuration, per action that defines the stable version of the library, ie:

          oozie.action.stable.library.version.streaming=0.20.204.0

          oozie.action.stable.library.version.pig=0.8.0

          * The ActionExecutor getShareLibPostFix() method would have the following logic:

          getShareLibPostFix(...) {

          String version = action.config.get("oozie.action.library.version");

          if (version == null) { bq. version = oozie.config.get("oozie.action.stable.library.version"); bq. }

          if (version != null) { bq. //construct the library path using the specified version bq. }

          }

          This approach does not require changes in the WF schema, instead uses a configuration value in case you want a different version than the stable, if not it just works for existing WF apps.

          If we are OK with this approach I'll file a JIRA with it.

          Mohammad Islam wrote:

          In addition to your proposal, we want to give the user the flexibility to choose the version optionally. User can specify that in wf.xml using use-version. Basically the effective version will be user-specified version , if it is defined in wf.xml. Otherwise, the default one that you mentioned will be the effective version.

          there is already a JIRA on this:

          https://issues.apache.org/jira/browse/OOZIE-555

          You can add your comments there.

          Alejandro Abdelnur wrote:

          In my proposal, if the user sets the configuration property 'oozie.action.library.version' to any value, that version will be used. If the user does not set it the default stable one will be used.

          I'll put my proposal in OOZIE-555

          Mohammad Islam wrote:

          The end-user who writes the workflow.xml should have a way of controlling his version. For example, one user tested his pig script againist pig 0.8 another one has used the latest (0.9). We should allow the both users to support that provide pig 0.8 and pig 0.9 are supported versions. I think it should be defined per-action level.

          Also we had the same design discussion few months back.

          yes, it is per action, the 'oozie.action.library.version' property is in the config section of each action, you can get it set to pig 0.8 in one action and to pig 0.9 in another. I've updated OOZIE-555 accordingly to this discussion. thxs

          • Alejandro

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2909/#review3446
          -----------------------------------------------------------

          On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2909/

          -----------------------------------------------------------

          (Updated 2011-11-22 21:03:19)

          Review request for oozie.

          Summary

          -------

          breaks the Oozie sharelib into one subdir per action type

          This addresses bug OOZIE-610.

          https://issues.apache.org/jira/browse/OOZIE-610

          Diffs

          -----

          /trunk/core/pom.xml 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165

          /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165

          /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165

          /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165

          /trunk/examples/pom.xml 1205165

          /trunk/examples/src/main/apps/custom-main/job.properties 1205165

          /trunk/examples/src/main/apps/demo/job.properties 1205165

          /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165

          /trunk/examples/src/main/apps/java-main/job.properties 1205165

          /trunk/examples/src/main/apps/pig/job.properties 1205165

          /trunk/examples/src/main/apps/streaming/job.properties 1205165

          /trunk/pom.xml 1205165

          /trunk/sharelib/pig/pom.xml PRE-CREATION

          /trunk/sharelib/pom.xml 1205165

          /trunk/sharelib/streaming/pom.xml PRE-CREATION

          /trunk/src/main/assemblies/distro.xml 1205165

          /trunk/src/main/assemblies/examples.xml 1205165

          /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION

          /trunk/src/main/assemblies/sharelib.xml 1205165

          Diff: https://reviews.apache.org/r/2909/diff

          Testing

          -------

          built, installed and run pig and streaming examples

          Thanks,

          Alejandro

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - On 2011-11-22 23:26:39, Mohammad Islam wrote: > Overall, it's an important problem. > > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars. > > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support. > > Alejandro Abdelnur wrote: Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected. Adding multi version support is a good idea. It would also require changes in the action XML to specify the version. Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later. Alejandro Abdelnur wrote: I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal. Thanks. Mohammad Islam wrote: I think the "use_version" support is considered here. Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2 Please review this patch. We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system) https://issues.apache.org/jira/browse/OOZIE-555 Alejandro Abdelnur wrote: Mmmh (regarding my lost comments). OK, the idea is similar, the main difference with what I'm proposing is that in Virag's patch it requires a new WF schema and that the library paths are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes that make difficult to see what is a change and what is not) To achieve similar functionality on top of OOZIE-610 we could: * Change the sharelib path to contain action library versions, ie: share/lib/streaming/0.20.204.0/ share/lib/pig/0.8.0/ share/lib/pig/0.9.0/ * Have an oozie-site configuration, per action that defines the stable version of the library, ie: oozie.action.stable.library.version.streaming=0.20.204.0 oozie.action.stable.library.version.pig=0.8.0 * The ActionExecutor getShareLibPostFix() method would have the following logic: getShareLibPostFix(...) { String version = action.config.get("oozie.action.library.version"); if (version == null) { bq. version = oozie.config.get("oozie.action.stable.library.version"); bq. } if (version != null) { bq. //construct the library path using the specified version bq. } } This approach does not require changes in the WF schema, instead uses a configuration value in case you want a different version than the stable, if not it just works for existing WF apps. If we are OK with this approach I'll file a JIRA with it. Mohammad Islam wrote: In addition to your proposal, we want to give the user the flexibility to choose the version optionally . User can specify that in wf.xml using use-version. Basically the effective version will be user-specified version , if it is defined in wf.xml. Otherwise, the default one that you mentioned will be the effective version. there is already a JIRA on this: https://issues.apache.org/jira/browse/OOZIE-555 You can add your comments there. Alejandro Abdelnur wrote: In my proposal, if the user sets the configuration property 'oozie.action.library.version' to any value, that version will be used. If the user does not set it the default stable one will be used. I'll put my proposal in OOZIE-555 Mohammad Islam wrote: The end-user who writes the workflow.xml should have a way of controlling his version. For example, one user tested his pig script againist pig 0.8 another one has used the latest (0.9). We should allow the both users to support that provide pig 0.8 and pig 0.9 are supported versions. I think it should be defined per-action level. Also we had the same design discussion few months back. yes, it is per action, the 'oozie.action.library.version' property is in the config section of each action, you can get it set to pig 0.8 in one action and to pig 0.9 in another. I've updated OOZIE-555 accordingly to this discussion. thxs Alejandro ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/#review3446 ----------------------------------------------------------- On 2011-11-22 21:03:19, Alejandro Abdelnur wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/ ----------------------------------------------------------- (Updated 2011-11-22 21:03:19) Review request for oozie. Summary ------- breaks the Oozie sharelib into one subdir per action type This addresses bug OOZIE-610 . https://issues.apache.org/jira/browse/OOZIE-610 Diffs ----- /trunk/core/pom.xml 1205165 /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165 /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165 /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165 /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165 /trunk/examples/pom.xml 1205165 /trunk/examples/src/main/apps/custom-main/job.properties 1205165 /trunk/examples/src/main/apps/demo/job.properties 1205165 /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165 /trunk/examples/src/main/apps/java-main/job.properties 1205165 /trunk/examples/src/main/apps/pig/job.properties 1205165 /trunk/examples/src/main/apps/streaming/job.properties 1205165 /trunk/pom.xml 1205165 /trunk/sharelib/pig/pom.xml PRE-CREATION /trunk/sharelib/pom.xml 1205165 /trunk/sharelib/streaming/pom.xml PRE-CREATION /trunk/src/main/assemblies/distro.xml 1205165 /trunk/src/main/assemblies/examples.xml 1205165 /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION /trunk/src/main/assemblies/sharelib.xml 1205165 Diff: https://reviews.apache.org/r/2909/diff Testing ------- built, installed and run pig and streaming examples Thanks, Alejandro
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2909/#review3490
          -----------------------------------------------------------

          Ship it!

          +1

          • Mohammad

          On 2011-11-23 21:45:14, Alejandro Abdelnur wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2909/

          -----------------------------------------------------------

          (Updated 2011-11-23 21:45:14)

          Review request for oozie.

          Summary

          -------

          breaks the Oozie sharelib into one subdir per action type

          This addresses bug OOZIE-610.

          https://issues.apache.org/jira/browse/OOZIE-610

          Diffs

          -----

          /trunk/core/pom.xml 1205616

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205616

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205616

          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205616

          /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205616

          /trunk/docs/src/site/twiki/DG_Examples.twiki 1205616

          /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205616

          /trunk/examples/pom.xml 1205616

          /trunk/examples/src/main/apps/custom-main/job.properties 1205616

          /trunk/examples/src/main/apps/demo/job.properties 1205616

          /trunk/examples/src/main/apps/hadoop-el/job.properties 1205616

          /trunk/examples/src/main/apps/java-main/job.properties 1205616

          /trunk/examples/src/main/apps/pig/job.properties 1205616

          /trunk/examples/src/main/apps/streaming/job.properties 1205616

          /trunk/pom.xml 1205616

          /trunk/sharelib/pig/pom.xml PRE-CREATION

          /trunk/sharelib/pom.xml 1205616

          /trunk/sharelib/streaming/pom.xml PRE-CREATION

          /trunk/src/main/assemblies/distro.xml 1205616

          /trunk/src/main/assemblies/examples.xml 1205616

          /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION

          /trunk/src/main/assemblies/sharelib.xml 1205616

          Diff: https://reviews.apache.org/r/2909/diff

          Testing

          -------

          built, installed and run pig and streaming examples

          Thanks,

          Alejandro

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/#review3490 ----------------------------------------------------------- Ship it! +1 Mohammad On 2011-11-23 21:45:14, Alejandro Abdelnur wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/ ----------------------------------------------------------- (Updated 2011-11-23 21:45:14) Review request for oozie. Summary ------- breaks the Oozie sharelib into one subdir per action type This addresses bug OOZIE-610 . https://issues.apache.org/jira/browse/OOZIE-610 Diffs ----- /trunk/core/pom.xml 1205616 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205616 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205616 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205616 /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205616 /trunk/docs/src/site/twiki/DG_Examples.twiki 1205616 /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205616 /trunk/examples/pom.xml 1205616 /trunk/examples/src/main/apps/custom-main/job.properties 1205616 /trunk/examples/src/main/apps/demo/job.properties 1205616 /trunk/examples/src/main/apps/hadoop-el/job.properties 1205616 /trunk/examples/src/main/apps/java-main/job.properties 1205616 /trunk/examples/src/main/apps/pig/job.properties 1205616 /trunk/examples/src/main/apps/streaming/job.properties 1205616 /trunk/pom.xml 1205616 /trunk/sharelib/pig/pom.xml PRE-CREATION /trunk/sharelib/pom.xml 1205616 /trunk/sharelib/streaming/pom.xml PRE-CREATION /trunk/src/main/assemblies/distro.xml 1205616 /trunk/src/main/assemblies/examples.xml 1205616 /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION /trunk/src/main/assemblies/sharelib.xml 1205616 Diff: https://reviews.apache.org/r/2909/diff Testing ------- built, installed and run pig and streaming examples Thanks, Alejandro
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2909/
          -----------------------------------------------------------

          (Updated 2011-11-23 21:45:14.340237)

          Review request for oozie.

          Changes
          -------

          removing spurious import from ActionExecutor.java

          Summary
          -------

          breaks the Oozie sharelib into one subdir per action type

          This addresses bug OOZIE-610.
          https://issues.apache.org/jira/browse/OOZIE-610

          Diffs (updated)


          /trunk/core/pom.xml 1205616
          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205616
          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205616
          /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205616
          /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205616
          /trunk/docs/src/site/twiki/DG_Examples.twiki 1205616
          /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205616
          /trunk/examples/pom.xml 1205616
          /trunk/examples/src/main/apps/custom-main/job.properties 1205616
          /trunk/examples/src/main/apps/demo/job.properties 1205616
          /trunk/examples/src/main/apps/hadoop-el/job.properties 1205616
          /trunk/examples/src/main/apps/java-main/job.properties 1205616
          /trunk/examples/src/main/apps/pig/job.properties 1205616
          /trunk/examples/src/main/apps/streaming/job.properties 1205616
          /trunk/pom.xml 1205616
          /trunk/sharelib/pig/pom.xml PRE-CREATION
          /trunk/sharelib/pom.xml 1205616
          /trunk/sharelib/streaming/pom.xml PRE-CREATION
          /trunk/src/main/assemblies/distro.xml 1205616
          /trunk/src/main/assemblies/examples.xml 1205616
          /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
          /trunk/src/main/assemblies/sharelib.xml 1205616

          Diff: https://reviews.apache.org/r/2909/diff

          Testing
          -------

          built, installed and run pig and streaming examples

          Thanks,

          Alejandro

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2909/ ----------------------------------------------------------- (Updated 2011-11-23 21:45:14.340237) Review request for oozie. Changes ------- removing spurious import from ActionExecutor.java Summary ------- breaks the Oozie sharelib into one subdir per action type This addresses bug OOZIE-610 . https://issues.apache.org/jira/browse/OOZIE-610 Diffs (updated) /trunk/core/pom.xml 1205616 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205616 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205616 /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205616 /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205616 /trunk/docs/src/site/twiki/DG_Examples.twiki 1205616 /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205616 /trunk/examples/pom.xml 1205616 /trunk/examples/src/main/apps/custom-main/job.properties 1205616 /trunk/examples/src/main/apps/demo/job.properties 1205616 /trunk/examples/src/main/apps/hadoop-el/job.properties 1205616 /trunk/examples/src/main/apps/java-main/job.properties 1205616 /trunk/examples/src/main/apps/pig/job.properties 1205616 /trunk/examples/src/main/apps/streaming/job.properties 1205616 /trunk/pom.xml 1205616 /trunk/sharelib/pig/pom.xml PRE-CREATION /trunk/sharelib/pom.xml 1205616 /trunk/sharelib/streaming/pom.xml PRE-CREATION /trunk/src/main/assemblies/distro.xml 1205616 /trunk/src/main/assemblies/examples.xml 1205616 /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION /trunk/src/main/assemblies/sharelib.xml 1205616 Diff: https://reviews.apache.org/r/2909/diff Testing ------- built, installed and run pig and streaming examples Thanks, Alejandro

            People

            • Assignee:
              tucu00 Alejandro Abdelnur
              Reporter:
              tucu00 Alejandro Abdelnur
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development