Oozie
  1. Oozie
  2. OOZIE-477

Adding configurable filesystem support instead of hardcoded "hdfs"

    Details

      Description

      This allows for configurable filesystem support in Hadoop 23 and not only "hdfs://".

      1. OOZIE-477v1.patch
        29 kB
        Mona Chitnis
      2. OOZIE-477v2.patch
        22 kB
        Mona Chitnis
      3. OOZIE-477v3.patch
        22 kB
        Mona Chitnis
      4. OOZIE-477v4.patch
        23 kB
        Mona Chitnis
      5. OOZIE-477v4rebasedtrunk.patch
        22 kB
        Mona Chitnis
      6. OOZIE-477v5.patch
        24 kB
        Mona Chitnis
      7. OOZIE-477v5rebased.patch
        24 kB
        Mona Chitnis
      8. OOZIE-477v8.patch
        28 kB
        Mona Chitnis

        Issue Links

          Activity

          Hide
          Jeremy Hanna added a comment -

          Since this includes support for filesystems other than hdfs (why was there a check for this in the first place?) my company would like to see that part get integrated.

          Show
          Jeremy Hanna added a comment - Since this includes support for filesystems other than hdfs (why was there a check for this in the first place?) my company would like to see that part get integrated.
          Hide
          Alejandro Abdelnur added a comment -

          The check was to avoid people running WFs from Oozie server local file system.

          Show
          Alejandro Abdelnur added a comment - The check was to avoid people running WFs from Oozie server local file system.
          Hide
          Jeremy Hanna added a comment -

          Ah - makes sense.

          Show
          Jeremy Hanna added a comment - Ah - makes sense.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/
          -----------------------------------------------------------

          (Updated 2012-02-08 20:02:12.539945)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Summary
          -------

          This patch was originally worked on by Mayank Bansal.
          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.
          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs


          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1175987
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1175987
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1175987
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1175987
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1175987
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/PipesMain.java 1175987
          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1175987
          trunk/core/src/main/resources/oozie-default.xml 1175987
          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1175987

          Diff: https://reviews.apache.org/r/2059/diff

          Testing
          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-02-08 20:02:12.539945) Review request for oozie, Mohammad Islam and Angelo K. Huang. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1175987 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1175987 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1175987 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1175987 trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1175987 trunk/core/src/main/java/org/apache/oozie/action/hadoop/PipesMain.java 1175987 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1175987 trunk/core/src/main/resources/oozie-default.xml 1175987 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1175987 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          Alejandro Abdelnur added a comment -

          The subject of the JIRA and the RB PATCH do not seem to match in what the do.

          I'll close this JIRA and discard the the RB request (https://reviews.apache.org/r/2059/).

          If the patch is still relevant/required, a new JIRA should be opened

          Show
          Alejandro Abdelnur added a comment - The subject of the JIRA and the RB PATCH do not seem to match in what the do. I'll close this JIRA and discard the the RB request ( https://reviews.apache.org/r/2059/ ). If the patch is still relevant/required, a new JIRA should be opened
          Hide
          Mona Chitnis added a comment -

          This patch specifically targets support in hadoop 23 for filesystems other than hdfs. I will reopen another JIRA with the relevant title.

          Thanks.

          Show
          Mona Chitnis added a comment - This patch specifically targets support in hadoop 23 for filesystems other than hdfs. I will reopen another JIRA with the relevant title. Thanks.
          Hide
          Jeremy Hanna added a comment -

          Mona: +1 - would be very useful for us.

          Show
          Jeremy Hanna added a comment - Mona: +1 - would be very useful for us.
          Hide
          Alejandro Abdelnur added a comment -

          Mona, you can edit the subject of this JIRA.

          Show
          Alejandro Abdelnur added a comment - Mona, you can edit the subject of this JIRA.
          Hide
          Mona Chitnis added a comment -

          Mayank,

          Since you are the reporter of this JIRA, would you please edit the title to appropriately describe your changes?

          Show
          Mona Chitnis added a comment - Mayank, Since you are the reporter of this JIRA, would you please edit the title to appropriately describe your changes?
          Hide
          Jeremy Hanna added a comment -

          Created OOZIE-695.

          Show
          Jeremy Hanna added a comment - Created OOZIE-695 .
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/
          -----------------------------------------------------------

          (Updated 2012-02-24 00:28:43.339663)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Summary
          -------

          This patch was originally worked on by Mayank Bansal.
          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.
          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs


          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1175987
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1175987
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1175987
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1175987
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1175987
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/PipesMain.java 1175987
          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1175987
          trunk/core/src/main/resources/oozie-default.xml 1175987
          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1175987

          Diff: https://reviews.apache.org/r/2059/diff

          Testing
          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-02-24 00:28:43.339663) Review request for oozie, Mohammad Islam and Angelo K. Huang. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1175987 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1175987 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1175987 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1175987 trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1175987 trunk/core/src/main/java/org/apache/oozie/action/hadoop/PipesMain.java 1175987 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1175987 trunk/core/src/main/resources/oozie-default.xml 1175987 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1175987 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          Jeremy Hanna added a comment -

          Can this ticket be re-opened to be able to review/commit with the patches that make it configurable. Alternately, could the hardcoding just be removed? I would go with either, but still not sure why it's a requirement. I think an argument could be made to throw an error when the prefix is 'file://' but defaulting to only allow hdfs is odd, even if it is configurable.

          Show
          Jeremy Hanna added a comment - Can this ticket be re-opened to be able to review/commit with the patches that make it configurable. Alternately, could the hardcoding just be removed? I would go with either, but still not sure why it's a requirement. I think an argument could be made to throw an error when the prefix is 'file://' but defaulting to only allow hdfs is odd, even if it is configurable.
          Hide
          Mona Chitnis added a comment -

          This patch will be committed to branch 3.1.4 as well as 3.2. I have put it up for review, on behalf of Mayank Bansal, on ReviewBoard - see link in comment before.

          Thanks

          Show
          Mona Chitnis added a comment - This patch will be committed to branch 3.1.4 as well as 3.2. I have put it up for review, on behalf of Mayank Bansal, on ReviewBoard - see link in comment before. Thanks
          Hide
          Mona Chitnis added a comment -

          Uploading rebased patch in some time to first cleanly apply to 3.1 (3.1.4)

          Show
          Mona Chitnis added a comment - Uploading rebased patch in some time to first cleanly apply to 3.1 (3.1.4)
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/
          -----------------------------------------------------------

          (Updated 2012-03-09 02:36:56.083797)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Changes
          -------

          rebased after trunk update to apply cleanly

          Summary
          -------

          This patch was originally worked on by Mayank Bansal.
          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.
          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs (updated)


          trunk/core/src/main/resources/oozie-default.xml 1298499
          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1298499
          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1298499
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/PipesMain.java 1298499
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1298499
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1298499
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1298499
          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1298499
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1298499

          Diff: https://reviews.apache.org/r/2059/diff

          Testing
          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-09 02:36:56.083797) Review request for oozie, Mohammad Islam and Angelo K. Huang. Changes ------- rebased after trunk update to apply cleanly Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs (updated) trunk/core/src/main/resources/oozie-default.xml 1298499 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1298499 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1298499 trunk/core/src/main/java/org/apache/oozie/action/hadoop/PipesMain.java 1298499 trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1298499 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1298499 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1298499 trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1298499 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1298499 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          Jeremy Hanna added a comment -

          Trivial nits:
          "Checking for supoorted Filesystem" should be "Checking for supported Filesystem"

          In getMaxExternalStatsSize, there is an updated instance, but the old one is left there but commented out:

          	Configuration conf = new Configuration();
                  //Configuration conf = new XConfiguration();
          

          also does this patch need the oozie.supported.multicluster property? if so, the text description needs to be fixed.

          All trivial nits - I'm not as familiar with the code but trying to help.

          Fwiw - the patch appeared to work fine when I tried to use it with an alternate DFS.

          Show
          Jeremy Hanna added a comment - Trivial nits: "Checking for supoorted Filesystem" should be "Checking for supported Filesystem" In getMaxExternalStatsSize, there is an updated instance, but the old one is left there but commented out: Configuration conf = new Configuration(); //Configuration conf = new XConfiguration(); also does this patch need the oozie.supported.multicluster property? if so, the text description needs to be fixed. All trivial nits - I'm not as familiar with the code but trying to help. Fwiw - the patch appeared to work fine when I tried to use it with an alternate DFS.
          Hide
          Mona Chitnis added a comment -

          Jeremy,

          I will incorporate your reviews. The patch does make use of the oozie.supported.multicluster property. Which text description are you referring to?

          Thanks for testing it out on your side.

          Regards,
          Mona

          Show
          Mona Chitnis added a comment - Jeremy, I will incorporate your reviews. The patch does make use of the oozie.supported.multicluster property. Which text description are you referring to? Thanks for testing it out on your side. Regards, Mona
          Hide
          Jeremy Hanna added a comment -

          the property description is in oozie-default.xml and reads:
          "Need to set it to 'on' if Ozzie suppose to work for multi namenode environment."

          Ozzie should be Oozie and might alternatively read:
          "Set to 'on' if Oozie should work in a multi-namenode environment."

          Show
          Jeremy Hanna added a comment - the property description is in oozie-default.xml and reads: "Need to set it to 'on' if Ozzie suppose to work for multi namenode environment." Ozzie should be Oozie and might alternatively read: "Set to 'on' if Oozie should work in a multi-namenode environment."
          Hide
          Jeremy Hanna added a comment -

          Re testing: I tested out the alternate filesystem support from a patch that I worked out of the original patches - I haven't had a chance to test out this particular subset but I would think they're pretty much the same.

          Show
          Jeremy Hanna added a comment - Re testing: I tested out the alternate filesystem support from a patch that I worked out of the original patches - I haven't had a chance to test out this particular subset but I would think they're pretty much the same.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/#review6087
          -----------------------------------------------------------

          Overall pls check the oozie formatting. Looks like there are some changes that are not related to this.

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment13062>

          We can break the loop here.

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment13063>

          remove this line.

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment13065>

          This could be a boolean value that will remove the "SUPPORT_MULTICLUSTER_ON" variable

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment13064>

          Why are we hard-coding the "JavaActionExecutor"?

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
          <https://reviews.apache.org/r/2059/#comment13066>

          why this many line? please use the oozie-formatting.

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
          <https://reviews.apache.org/r/2059/#comment13067>

          I think int he new code "group.name" is removed from all over.

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
          <https://reviews.apache.org/r/2059/#comment13068>

          same for 'group'

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
          <https://reviews.apache.org/r/2059/#comment13069>

          extra space

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment13070>

          formatting all over

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/PipesMain.java
          <https://reviews.apache.org/r/2059/#comment13071>

          why these new two imports?

          • Mohammad

          On 2012-03-09 02:36:56, Mona Chitnis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2059/

          -----------------------------------------------------------

          (Updated 2012-03-09 02:36:56)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Summary

          -------

          This patch was originally worked on by Mayank Bansal.

          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.

          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs

          -----

          trunk/core/src/main/resources/oozie-default.xml 1298499

          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1298499

          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1298499

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/PipesMain.java 1298499

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1298499

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1298499

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1298499

          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1298499

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1298499

          Diff: https://reviews.apache.org/r/2059/diff

          Testing

          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/#review6087 ----------------------------------------------------------- Overall pls check the oozie formatting. Looks like there are some changes that are not related to this. trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java < https://reviews.apache.org/r/2059/#comment13062 > We can break the loop here. trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java < https://reviews.apache.org/r/2059/#comment13063 > remove this line. trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java < https://reviews.apache.org/r/2059/#comment13065 > This could be a boolean value that will remove the "SUPPORT_MULTICLUSTER_ON" variable trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java < https://reviews.apache.org/r/2059/#comment13064 > Why are we hard-coding the "JavaActionExecutor"? trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java < https://reviews.apache.org/r/2059/#comment13066 > why this many line? please use the oozie-formatting. trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java < https://reviews.apache.org/r/2059/#comment13067 > I think int he new code "group.name" is removed from all over. trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java < https://reviews.apache.org/r/2059/#comment13068 > same for 'group' trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java < https://reviews.apache.org/r/2059/#comment13069 > extra space trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java < https://reviews.apache.org/r/2059/#comment13070 > formatting all over trunk/core/src/main/java/org/apache/oozie/action/hadoop/PipesMain.java < https://reviews.apache.org/r/2059/#comment13071 > why these new two imports? Mohammad On 2012-03-09 02:36:56, Mona Chitnis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-09 02:36:56) Review request for oozie, Mohammad Islam and Angelo K. Huang. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs ----- trunk/core/src/main/resources/oozie-default.xml 1298499 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1298499 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1298499 trunk/core/src/main/java/org/apache/oozie/action/hadoop/PipesMain.java 1298499 trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1298499 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1298499 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1298499 trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1298499 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1298499 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/
          -----------------------------------------------------------

          (Updated 2012-03-22 08:00:43.598561)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Changes
          -------

          incorporated review feedback

          Summary
          -------

          This patch was originally worked on by Mayank Bansal.
          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.
          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs (updated)


          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1303609
          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609
          trunk/core/src/main/resources/oozie-default.xml 1303609
          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing
          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-22 08:00:43.598561) Review request for oozie, Mohammad Islam and Angelo K. Huang. Changes ------- incorporated review feedback Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs (updated) trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          Jeremy Hanna added a comment -

          Will this be able to make it into 3.2?

          Show
          Jeremy Hanna added a comment - Will this be able to make it into 3.2?
          Hide
          Mona Chitnis added a comment -

          yes most likely

          Show
          Mona Chitnis added a comment - yes most likely
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/
          -----------------------------------------------------------

          (Updated 2012-03-22 17:27:37.492001)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Changes
          -------

          reverted change to oozie.supported.multicluster property=false in oozie-default.xml

          Summary
          -------

          This patch was originally worked on by Mayank Bansal.
          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.
          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs (updated)


          trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1303609
          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609
          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609
          trunk/core/src/main/resources/oozie-default.xml 1303609
          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing
          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-22 17:27:37.492001) Review request for oozie, Mohammad Islam and Angelo K. Huang. Changes ------- reverted change to oozie.supported.multicluster property=false in oozie-default.xml Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs (updated) trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1303609 trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/#review6307
          -----------------------------------------------------------

          the FSActionExecutor is checking for the allowed filesystem schemes. It seems the JavaActionExecutor and the submit job command should do the same, no?

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment13691>

          we should read/parse/trim the supported filesystem schemes once (at init time, or first time) and then cache the array

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment13695>

          what is this multicluster safeguard check? don't understand what does it prevent/enable

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment13694>

          where is this method used?

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment13687>

          this is a non-relevant change. please revert

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment13688>

          same same

          • Alejandro

          On 2012-03-22 17:27:37, Mona Chitnis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2059/

          -----------------------------------------------------------

          (Updated 2012-03-22 17:27:37)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Summary

          -------

          This patch was originally worked on by Mayank Bansal.

          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.

          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs

          -----

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1303609

          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609

          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609

          trunk/core/src/main/resources/oozie-default.xml 1303609

          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing

          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/#review6307 ----------------------------------------------------------- the FSActionExecutor is checking for the allowed filesystem schemes. It seems the JavaActionExecutor and the submit job command should do the same, no? trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java < https://reviews.apache.org/r/2059/#comment13691 > we should read/parse/trim the supported filesystem schemes once (at init time, or first time) and then cache the array trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java < https://reviews.apache.org/r/2059/#comment13695 > what is this multicluster safeguard check? don't understand what does it prevent/enable trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java < https://reviews.apache.org/r/2059/#comment13694 > where is this method used? trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java < https://reviews.apache.org/r/2059/#comment13687 > this is a non-relevant change. please revert trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java < https://reviews.apache.org/r/2059/#comment13688 > same same Alejandro On 2012-03-22 17:27:37, Mona Chitnis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-22 17:27:37) Review request for oozie, Mohammad Islam and Angelo K. Huang. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs ----- trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1303609 trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          Mona Chitnis added a comment -

          Alejandro,

          Thanks for your review. I will look into incorporating/commenting on it so we can proceed. In the meanwhile, also asking Mayank to chip in with his views.

          Show
          Mona Chitnis added a comment - Alejandro, Thanks for your review. I will look into incorporating/commenting on it so we can proceed. In the meanwhile, also asking Mayank to chip in with his views.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/#review6321
          -----------------------------------------------------------

          Ship it!

          +1

          • mayank

          On 2012-03-22 17:27:37, Mona Chitnis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2059/

          -----------------------------------------------------------

          (Updated 2012-03-22 17:27:37)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Summary

          -------

          This patch was originally worked on by Mayank Bansal.

          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.

          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs

          -----

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1303609

          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609

          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609

          trunk/core/src/main/resources/oozie-default.xml 1303609

          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing

          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/#review6321 ----------------------------------------------------------- Ship it! +1 mayank On 2012-03-22 17:27:37, Mona Chitnis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-22 17:27:37) Review request for oozie, Mohammad Islam and Angelo K. Huang. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs ----- trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1303609 trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/#review6323
          -----------------------------------------------------------

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment13738>

          Remove this method

          • mayank

          On 2012-03-22 17:27:37, Mona Chitnis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2059/

          -----------------------------------------------------------

          (Updated 2012-03-22 17:27:37)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Summary

          -------

          This patch was originally worked on by Mayank Bansal.

          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.

          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs

          -----

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1303609

          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609

          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609

          trunk/core/src/main/resources/oozie-default.xml 1303609

          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing

          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/#review6323 ----------------------------------------------------------- trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java < https://reviews.apache.org/r/2059/#comment13738 > Remove this method mayank On 2012-03-22 17:27:37, Mona Chitnis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-22 17:27:37) Review request for oozie, Mohammad Islam and Angelo K. Huang. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs ----- trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1303609 trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-03-24 04:26:38, mayank wrote:

          > +1

          Can we please hold on committing this, I'd like my comments be answered.

          Thxs.

          Alejandro

          • Alejandro

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/#review6321
          -----------------------------------------------------------

          On 2012-03-22 17:27:37, Mona Chitnis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2059/

          -----------------------------------------------------------

          (Updated 2012-03-22 17:27:37)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Summary

          -------

          This patch was originally worked on by Mayank Bansal.

          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.

          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs

          -----

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1303609

          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609

          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609

          trunk/core/src/main/resources/oozie-default.xml 1303609

          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing

          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-03-24 04:26:38, mayank wrote: > +1 Can we please hold on committing this, I'd like my comments be answered. Thxs. Alejandro Alejandro ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/#review6321 ----------------------------------------------------------- On 2012-03-22 17:27:37, Mona Chitnis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-22 17:27:37) Review request for oozie, Mohammad Islam and Angelo K. Huang. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs ----- trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1303609 trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/#review6332
          -----------------------------------------------------------

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment13756>

          This requirement came up when Mayank tested for multiple NN (federation) support. Without that , multiple NN access was not working. I remember Sanjay Radia proposed to follow this approach to get the URI.

          However it was a very long time back.

          • Mohammad

          On 2012-03-22 17:27:37, Mona Chitnis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2059/

          -----------------------------------------------------------

          (Updated 2012-03-22 17:27:37)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Summary

          -------

          This patch was originally worked on by Mayank Bansal.

          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.

          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs

          -----

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1303609

          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609

          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609

          trunk/core/src/main/resources/oozie-default.xml 1303609

          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing

          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/#review6332 ----------------------------------------------------------- trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java < https://reviews.apache.org/r/2059/#comment13756 > This requirement came up when Mayank tested for multiple NN (federation) support. Without that , multiple NN access was not working. I remember Sanjay Radia proposed to follow this approach to get the URI. However it was a very long time back. Mohammad On 2012-03-22 17:27:37, Mona Chitnis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-22 17:27:37) Review request for oozie, Mohammad Islam and Angelo K. Huang. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs ----- trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1303609 trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-03-23 20:46:30, Alejandro Abdelnur wrote:

          > the FSActionExecutor is checking for the allowed filesystem schemes. It seems the JavaActionExecutor and the submit job command should do the same, no?

          In earlier code, the FsActionExecutor had hardcoding of "hdfs://" scheme. This patch is mainly directed to remove that hardcoding and allow flexibility in schemes, through configuration. There can be other places too, in addition to JavaActionExecutor where code change is required for federation. I am opening another JIRA to address that requirement.

          On 2012-03-23 20:46:30, Alejandro Abdelnur wrote:

          > trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java, line 97

          > <https://reviews.apache.org/r/2059/diff/6/?file=94571#file94571line97>

          >

          > this is a non-relevant change. please revert

          ok

          On 2012-03-23 20:46:30, Alejandro Abdelnur wrote:

          > trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java, line 102

          > <https://reviews.apache.org/r/2059/diff/6/?file=94571#file94571line102>

          >

          > same same

          ok

          On 2012-03-23 20:46:30, Alejandro Abdelnur wrote:

          > trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java, line 98

          > <https://reviews.apache.org/r/2059/diff/6/?file=94568#file94568line98>

          >

          > we should read/parse/trim the supported filesystem schemes once (at init time, or first time) and then cache the array

          agreed. otherwise the config property - supported schemes, is being retrieved repeatedly. fixed.

          On 2012-03-23 20:46:30, Alejandro Abdelnur wrote:

          > trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1114

          > <https://reviews.apache.org/r/2059/diff/6/?file=94569#file94569line1114>

          >

          > where is this method used?

          removing method since it is not used.

          • Mona

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/#review6307
          -----------------------------------------------------------

          On 2012-03-22 17:27:37, Mona Chitnis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2059/

          -----------------------------------------------------------

          (Updated 2012-03-22 17:27:37)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Summary

          -------

          This patch was originally worked on by Mayank Bansal.

          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.

          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs

          -----

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1303609

          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609

          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609

          trunk/core/src/main/resources/oozie-default.xml 1303609

          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing

          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-03-23 20:46:30, Alejandro Abdelnur wrote: > the FSActionExecutor is checking for the allowed filesystem schemes. It seems the JavaActionExecutor and the submit job command should do the same, no? In earlier code, the FsActionExecutor had hardcoding of "hdfs://" scheme. This patch is mainly directed to remove that hardcoding and allow flexibility in schemes, through configuration. There can be other places too, in addition to JavaActionExecutor where code change is required for federation. I am opening another JIRA to address that requirement. On 2012-03-23 20:46:30, Alejandro Abdelnur wrote: > trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java, line 97 > < https://reviews.apache.org/r/2059/diff/6/?file=94571#file94571line97 > > > this is a non-relevant change. please revert ok On 2012-03-23 20:46:30, Alejandro Abdelnur wrote: > trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java, line 102 > < https://reviews.apache.org/r/2059/diff/6/?file=94571#file94571line102 > > > same same ok On 2012-03-23 20:46:30, Alejandro Abdelnur wrote: > trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java, line 98 > < https://reviews.apache.org/r/2059/diff/6/?file=94568#file94568line98 > > > we should read/parse/trim the supported filesystem schemes once (at init time, or first time) and then cache the array agreed. otherwise the config property - supported schemes, is being retrieved repeatedly. fixed. On 2012-03-23 20:46:30, Alejandro Abdelnur wrote: > trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1114 > < https://reviews.apache.org/r/2059/diff/6/?file=94569#file94569line1114 > > > where is this method used? removing method since it is not used. Mona ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/#review6307 ----------------------------------------------------------- On 2012-03-22 17:27:37, Mona Chitnis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-22 17:27:37) Review request for oozie, Mohammad Islam and Angelo K. Huang. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs ----- trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1303609 trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/
          -----------------------------------------------------------

          (Updated 2012-03-27 00:15:40.905318)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Changes
          -------

          addressed review feedback.

          Summary
          -------

          This patch was originally worked on by Mayank Bansal.
          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.
          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs (updated)


          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609
          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609
          trunk/core/src/main/resources/oozie-default.xml 1303609
          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing
          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-27 00:15:40.905318) Review request for oozie, Mohammad Islam and Angelo K. Huang. Changes ------- addressed review feedback. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs (updated) trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/#review6442
          -----------------------------------------------------------

          The check for supported FS schemes should also be done in the submission of the WF/COORD/BUNDLE job, no?

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment14064>

          this seems like a complete different thing from enabling different filesystem schemes. we should do this as part of a different JIRA

          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java
          <https://reviews.apache.org/r/2059/#comment14063>

          (String) casting is not required

          • Alejandro

          On 2012-03-27 00:15:40, Mona Chitnis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2059/

          -----------------------------------------------------------

          (Updated 2012-03-27 00:15:40)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Summary

          -------

          This patch was originally worked on by Mayank Bansal.

          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.

          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs

          -----

          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609

          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609

          trunk/core/src/main/resources/oozie-default.xml 1303609

          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing

          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/#review6442 ----------------------------------------------------------- The check for supported FS schemes should also be done in the submission of the WF/COORD/BUNDLE job, no? trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java < https://reviews.apache.org/r/2059/#comment14064 > this seems like a complete different thing from enabling different filesystem schemes. we should do this as part of a different JIRA trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java < https://reviews.apache.org/r/2059/#comment14063 > (String) casting is not required Alejandro On 2012-03-27 00:15:40, Mona Chitnis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-27 00:15:40) Review request for oozie, Mohammad Islam and Angelo K. Huang. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs ----- trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-03-27 20:14:21, Alejandro Abdelnur wrote:

          > The check for supported FS schemes should also be done in the submission of the WF/COORD/BUNDLE job, no?

          Yes thats right. As I replied to your earlier comment, there can be other places too, in addition to JavaActionExecutor where code change is required for federation. I am going to address that requirement in another JIRA OOZIE-782. This JIRA will be only to remove the hardcoding of "hdfs://"

          On 2012-03-27 20:14:21, Alejandro Abdelnur wrote:

          > trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1091

          > <https://reviews.apache.org/r/2059/diff/7/?file=96447#file96447line1091>

          >

          > this seems like a complete different thing from enabling different filesystem schemes. we should do this as part of a different JIRA

          Moving this out of this JIRA and into OOZIE-782

          On 2012-03-27 20:14:21, Alejandro Abdelnur wrote:

          > trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java, line 248

          > <https://reviews.apache.org/r/2059/diff/7/?file=96449#file96449line248>

          >

          > (String) casting is not required

          done

          • Mona

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/#review6442
          -----------------------------------------------------------

          On 2012-03-27 00:15:40, Mona Chitnis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2059/

          -----------------------------------------------------------

          (Updated 2012-03-27 00:15:40)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Summary

          -------

          This patch was originally worked on by Mayank Bansal.

          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.

          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs

          -----

          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609

          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609

          trunk/core/src/main/resources/oozie-default.xml 1303609

          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing

          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-03-27 20:14:21, Alejandro Abdelnur wrote: > The check for supported FS schemes should also be done in the submission of the WF/COORD/BUNDLE job, no? Yes thats right. As I replied to your earlier comment, there can be other places too, in addition to JavaActionExecutor where code change is required for federation. I am going to address that requirement in another JIRA OOZIE-782 . This JIRA will be only to remove the hardcoding of "hdfs://" On 2012-03-27 20:14:21, Alejandro Abdelnur wrote: > trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1091 > < https://reviews.apache.org/r/2059/diff/7/?file=96447#file96447line1091 > > > this seems like a complete different thing from enabling different filesystem schemes. we should do this as part of a different JIRA Moving this out of this JIRA and into OOZIE-782 On 2012-03-27 20:14:21, Alejandro Abdelnur wrote: > trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java, line 248 > < https://reviews.apache.org/r/2059/diff/7/?file=96449#file96449line248 > > > (String) casting is not required done Mona ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/#review6442 ----------------------------------------------------------- On 2012-03-27 00:15:40, Mona Chitnis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-27 00:15:40) Review request for oozie, Mohammad Islam and Angelo K. Huang. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs ----- trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/
          -----------------------------------------------------------

          (Updated 2012-03-28 00:14:46.724524)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Changes
          -------

          incorporated review feedback. Scope of JIRA made to only avoid "hdfs://" hardcoding. Checking supported schema for workflow/coord/bundle job submission in various action executors will be addressed in OOZIE-782

          Summary
          -------

          This patch was originally worked on by Mayank Bansal.
          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.
          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs (updated)


          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609
          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609
          trunk/core/src/main/resources/oozie-default.xml 1303609
          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing
          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-28 00:14:46.724524) Review request for oozie, Mohammad Islam and Angelo K. Huang. Changes ------- incorporated review feedback. Scope of JIRA made to only avoid "hdfs://" hardcoding. Checking supported schema for workflow/coord/bundle job submission in various action executors will be addressed in OOZIE-782 Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs (updated) trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/#review6487
          -----------------------------------------------------------

          I think my previous comment on the WFApp not checking the supported filesystems was not fully understood.

          We should check that the app path is for a supported filesystem and fail if not. else people could write WFs in the local filesystem or NFS and loaded from Oozie. This should be possible only if it is a supported filesystem

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment14136>

          this should be done once, if supportedSchemes is null. it should be done within a synchronized block:

          synch {
          if (supportedSchemes == null)

          { ///load }

          }

          this would create minimum contention as there is no logic at all after the first pass that loads the supported schemes

          • Alejandro

          On 2012-03-28 00:14:46, Mona Chitnis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2059/

          -----------------------------------------------------------

          (Updated 2012-03-28 00:14:46)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Summary

          -------

          This patch was originally worked on by Mayank Bansal.

          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.

          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs

          -----

          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609

          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609

          trunk/core/src/main/resources/oozie-default.xml 1303609

          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing

          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/#review6487 ----------------------------------------------------------- I think my previous comment on the WFApp not checking the supported filesystems was not fully understood. We should check that the app path is for a supported filesystem and fail if not. else people could write WFs in the local filesystem or NFS and loaded from Oozie. This should be possible only if it is a supported filesystem trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java < https://reviews.apache.org/r/2059/#comment14136 > this should be done once, if supportedSchemes is null. it should be done within a synchronized block: synch { if (supportedSchemes == null) { ///load } } this would create minimum contention as there is no logic at all after the first pass that loads the supported schemes Alejandro On 2012-03-28 00:14:46, Mona Chitnis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-28 00:14:46) Review request for oozie, Mohammad Islam and Angelo K. Huang. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs ----- trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-03-28 19:47:57, Alejandro Abdelnur wrote:

          > I think my previous comment on the WFApp not checking the supported filesystems was not fully understood.

          >

          > We should check that the app path is for a supported filesystem and fail if not. else people could write WFs in the local filesystem or NFS and loaded from Oozie. This should be possible only if it is a supported filesystem

          added the Wf app path check. To avoid code repetition, defined this logic of "checkSupportedFilesystem" in parent class ActionExecutor and invoking from JavaAE and FsAE. Modified/added respective testcases.

          On 2012-03-28 19:47:57, Alejandro Abdelnur wrote:

          > trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java, line 358

          > <https://reviews.apache.org/r/2059/diff/8/?file=97129#file97129line358>

          >

          > this should be done once, if supportedSchemes is null. it should be done within a synchronized block:

          >

          > synch {

          > if (supportedSchemes == null) { bq. > ///load bq. > }

          > }

          >

          > this would create minimum contention as there is no logic at all after the first pass that loads the supported schemes

          done

          • Mona

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/#review6487
          -----------------------------------------------------------

          On 2012-03-28 00:14:46, Mona Chitnis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2059/

          -----------------------------------------------------------

          (Updated 2012-03-28 00:14:46)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Summary

          -------

          This patch was originally worked on by Mayank Bansal.

          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.

          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs

          -----

          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609

          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609

          trunk/core/src/main/resources/oozie-default.xml 1303609

          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing

          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-03-28 19:47:57, Alejandro Abdelnur wrote: > I think my previous comment on the WFApp not checking the supported filesystems was not fully understood. > > We should check that the app path is for a supported filesystem and fail if not. else people could write WFs in the local filesystem or NFS and loaded from Oozie. This should be possible only if it is a supported filesystem added the Wf app path check. To avoid code repetition, defined this logic of "checkSupportedFilesystem" in parent class ActionExecutor and invoking from JavaAE and FsAE. Modified/added respective testcases. On 2012-03-28 19:47:57, Alejandro Abdelnur wrote: > trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java, line 358 > < https://reviews.apache.org/r/2059/diff/8/?file=97129#file97129line358 > > > this should be done once, if supportedSchemes is null. it should be done within a synchronized block: > > synch { > if (supportedSchemes == null) { bq. > ///load bq. > } > } > > this would create minimum contention as there is no logic at all after the first pass that loads the supported schemes done Mona ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/#review6487 ----------------------------------------------------------- On 2012-03-28 00:14:46, Mona Chitnis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-28 00:14:46) Review request for oozie, Mohammad Islam and Angelo K. Huang. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs ----- trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/
          -----------------------------------------------------------

          (Updated 2012-03-29 10:29:06.751615)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Changes
          -------

          review feedback addressed.

          Summary
          -------

          This patch was originally worked on by Mayank Bansal.
          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.
          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs (updated)


          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609
          trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1303609
          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609
          trunk/core/src/main/resources/oozie-default.xml 1303609
          trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFsActionExecutor.java 1303609
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609
          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609
          trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1303609
          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing
          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-29 10:29:06.751615) Review request for oozie, Mohammad Islam and Angelo K. Huang. Changes ------- review feedback addressed. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs (updated) trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609 trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/#review6513
          -----------------------------------------------------------

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment14168>

          wasn't this going to be addressed by another JIRA?

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment14169>

          the synchronization cannot be on the current action instance as actions are created every time they are needed. the synchronization should be on something global, ie the class hosting this method

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment14170>

          again, this is suppose to be part of another JIRA, no?

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
          <https://reviews.apache.org/r/2059/#comment14171>

          this code is duplicated in FSActionExecutor, it seems it should be factored out and reused.

          Furthermore, should this be loaded then by the ActionExecutor class itself?

          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java
          <https://reviews.apache.org/r/2059/#comment14172>

          what am i missing? i still don't see the scheme being check here.

          • Alejandro

          On 2012-03-29 10:29:06, Mona Chitnis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2059/

          -----------------------------------------------------------

          (Updated 2012-03-29 10:29:06)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Summary

          -------

          This patch was originally worked on by Mayank Bansal.

          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.

          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs

          -----

          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609

          trunk/core/src/main/resources/oozie-default.xml 1303609

          trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFsActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609

          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing

          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/#review6513 ----------------------------------------------------------- trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java < https://reviews.apache.org/r/2059/#comment14168 > wasn't this going to be addressed by another JIRA? trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java < https://reviews.apache.org/r/2059/#comment14169 > the synchronization cannot be on the current action instance as actions are created every time they are needed. the synchronization should be on something global, ie the class hosting this method trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java < https://reviews.apache.org/r/2059/#comment14170 > again, this is suppose to be part of another JIRA, no? trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java < https://reviews.apache.org/r/2059/#comment14171 > this code is duplicated in FSActionExecutor, it seems it should be factored out and reused. Furthermore, should this be loaded then by the ActionExecutor class itself? trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java < https://reviews.apache.org/r/2059/#comment14172 > what am i missing? i still don't see the scheme being check here. Alejandro On 2012-03-29 10:29:06, Mona Chitnis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-29 10:29:06) Review request for oozie, Mohammad Islam and Angelo K. Huang. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs ----- trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609 trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-03-29 10:42:44, Alejandro Abdelnur wrote:

          > trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java, line 334

          > <https://reviews.apache.org/r/2059/diff/9/?file=97335#file97335line334>

          >

          > the synchronization cannot be on the current action instance as actions are created every time they are needed. the synchronization should be on something global, ie the class hosting this method

          ok. fixing

          On 2012-03-29 10:42:44, Alejandro Abdelnur wrote:

          > trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java, line 341

          > <https://reviews.apache.org/r/2059/diff/9/?file=97335#file97335line341>

          >

          > again, this is suppose to be part of another JIRA, no?

          similar to above comment for this variable.

          On 2012-03-29 10:42:44, Alejandro Abdelnur wrote:

          > trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java, line 46

          > <https://reviews.apache.org/r/2059/diff/9/?file=97335#file97335line46>

          >

          > wasn't this going to be addressed by another JIRA?

          this safeguard was required against the case if there are blank entries for "oozie.filesystems.supported" in oozie-site.xml or other places to configure. In that case, variable 'supportedSchemes' in ActionExecutor will be "" and all schemes will fail at check.

          On 2012-03-29 10:42:44, Alejandro Abdelnur wrote:

          > trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 202

          > <https://reviews.apache.org/r/2059/diff/9/?file=97336#file97336line202>

          >

          > this code is duplicated in FSActionExecutor, it seems it should be factored out and reused.

          >

          > Furthermore, should this be loaded then by the ActionExecutor class itself?

          Checking if can be factored out up into ActionExecutor

          On 2012-03-29 10:42:44, Alejandro Abdelnur wrote:

          > trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java, line 153

          > <https://reviews.apache.org/r/2059/diff/9/?file=97338#file97338line153>

          >

          > what am i missing? i still don't see the scheme being check here.

          Wouldn't the checking in respective action executors suffice ?

          • Mona

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/#review6513
          -----------------------------------------------------------

          On 2012-03-29 10:29:06, Mona Chitnis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2059/

          -----------------------------------------------------------

          (Updated 2012-03-29 10:29:06)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Summary

          -------

          This patch was originally worked on by Mayank Bansal.

          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.

          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs

          -----

          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609

          trunk/core/src/main/resources/oozie-default.xml 1303609

          trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFsActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609

          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing

          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-03-29 10:42:44, Alejandro Abdelnur wrote: > trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java, line 334 > < https://reviews.apache.org/r/2059/diff/9/?file=97335#file97335line334 > > > the synchronization cannot be on the current action instance as actions are created every time they are needed. the synchronization should be on something global, ie the class hosting this method ok. fixing On 2012-03-29 10:42:44, Alejandro Abdelnur wrote: > trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java, line 341 > < https://reviews.apache.org/r/2059/diff/9/?file=97335#file97335line341 > > > again, this is suppose to be part of another JIRA, no? similar to above comment for this variable. On 2012-03-29 10:42:44, Alejandro Abdelnur wrote: > trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java, line 46 > < https://reviews.apache.org/r/2059/diff/9/?file=97335#file97335line46 > > > wasn't this going to be addressed by another JIRA? this safeguard was required against the case if there are blank entries for "oozie.filesystems.supported" in oozie-site.xml or other places to configure. In that case, variable 'supportedSchemes' in ActionExecutor will be "" and all schemes will fail at check. On 2012-03-29 10:42:44, Alejandro Abdelnur wrote: > trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 202 > < https://reviews.apache.org/r/2059/diff/9/?file=97336#file97336line202 > > > this code is duplicated in FSActionExecutor, it seems it should be factored out and reused. > > Furthermore, should this be loaded then by the ActionExecutor class itself? Checking if can be factored out up into ActionExecutor On 2012-03-29 10:42:44, Alejandro Abdelnur wrote: > trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java, line 153 > < https://reviews.apache.org/r/2059/diff/9/?file=97338#file97338line153 > > > what am i missing? i still don't see the scheme being check here. Wouldn't the checking in respective action executors suffice ? Mona ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/#review6513 ----------------------------------------------------------- On 2012-03-29 10:29:06, Mona Chitnis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-29 10:29:06) Review request for oozie, Mohammad Islam and Angelo K. Huang. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs ----- trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609 trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-03-29 10:42:44, Alejandro Abdelnur wrote:

          > trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java, line 153

          > <https://reviews.apache.org/r/2059/diff/9/?file=97338#file97338line153>

          >

          > what am i missing? i still don't see the scheme being check here.

          Mona Chitnis wrote:

          Wouldn't the checking in respective action executors suffice ?

          we should use the same scheme check to enforce where Oozie apps (XMLs) can be read from. Not only for WFs but for COORDs and BUNDLEs

          • Alejandro

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2059/#review6513
          -----------------------------------------------------------

          On 2012-03-29 10:29:06, Mona Chitnis wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2059/

          -----------------------------------------------------------

          (Updated 2012-03-29 10:29:06)

          Review request for oozie, Mohammad Islam and Angelo K. Huang.

          Summary

          -------

          This patch was originally worked on by Mayank Bansal.

          I (Mona Chitnis) am in charge of handling this review request.

          Description - adding hadoop.next support for oozie

          This addresses bug OOZIE-477.

          https://issues.apache.org/jira/browse/OOZIE-477

          Diffs

          -----

          trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609

          trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609

          trunk/core/src/main/resources/oozie-default.xml 1303609

          trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFsActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609

          trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1303609

          trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609

          Diff: https://reviews.apache.org/r/2059/diff

          Testing

          -------

          tested against current unit tests

          Thanks,

          Mona

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-03-29 10:42:44, Alejandro Abdelnur wrote: > trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java, line 153 > < https://reviews.apache.org/r/2059/diff/9/?file=97338#file97338line153 > > > what am i missing? i still don't see the scheme being check here. Mona Chitnis wrote: Wouldn't the checking in respective action executors suffice ? we should use the same scheme check to enforce where Oozie apps (XMLs) can be read from. Not only for WFs but for COORDs and BUNDLEs Alejandro ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/#review6513 ----------------------------------------------------------- On 2012-03-29 10:29:06, Mona Chitnis wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2059/ ----------------------------------------------------------- (Updated 2012-03-29 10:29:06) Review request for oozie, Mohammad Islam and Angelo K. Huang. Summary ------- This patch was originally worked on by Mayank Bansal. I (Mona Chitnis) am in charge of handling this review request. Description - adding hadoop.next support for oozie This addresses bug OOZIE-477 . https://issues.apache.org/jira/browse/OOZIE-477 Diffs ----- trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java 1303609 trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1303609 trunk/core/src/main/resources/oozie-default.xml 1303609 trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFsActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1303609 trunk/client/src/main/java/org/apache/oozie/client/XOozieClient.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1303609 trunk/core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 1303609 Diff: https://reviews.apache.org/r/2059/diff Testing ------- tested against current unit tests Thanks, Mona
          Hide
          Jeremy Hanna added a comment -

          I commented on the review board but not sure how it shows up again. I think the only real place that I saw in the codebase that has the hard coded check for hdfs was in the FSActionExecutor. In other words, with just that change and the addition of the configuration, alternative filesystems work correctly with oozie. So although it would be nice to have there be a more general ActionExecutor that has the check in there as well as checking in other places - like the paths in the workflow and coordinator configurations, the check is not currently done in oozie. Could that be a separate ticket - since it's in addition to the functionality described in this ticket? Also, I saw in the tests for FSActionExecutor that it tests "hdfs://" and "blah" prefixes. It would be nice to test the other prefix that is in the schemes that are in the test - viewfs - to assert that works.

          Show
          Jeremy Hanna added a comment - I commented on the review board but not sure how it shows up again. I think the only real place that I saw in the codebase that has the hard coded check for hdfs was in the FSActionExecutor. In other words, with just that change and the addition of the configuration, alternative filesystems work correctly with oozie. So although it would be nice to have there be a more general ActionExecutor that has the check in there as well as checking in other places - like the paths in the workflow and coordinator configurations, the check is not currently done in oozie. Could that be a separate ticket - since it's in addition to the functionality described in this ticket? Also, I saw in the tests for FSActionExecutor that it tests "hdfs://" and "blah" prefixes. It would be nice to test the other prefix that is in the schemes that are in the test - viewfs - to assert that works.
          Hide
          Jeremy Hanna added a comment -

          And I'm new to this codebase, so let me know if I'm not understanding something here .

          Show
          Jeremy Hanna added a comment - And I'm new to this codebase, so let me know if I'm not understanding something here .
          Hide
          Jeremy Hanna added a comment -

          Mona and Alejandro,

          Does that make sense and would that be reasonable to split into solving this ticket to make it possible, and then a follow-up ticket to make that checking pervasive?

          Show
          Jeremy Hanna added a comment - Mona and Alejandro, Does that make sense and would that be reasonable to split into solving this ticket to make it possible, and then a follow-up ticket to make that checking pervasive?
          Hide
          Mona Chitnis added a comment -

          I agree that the recent changes in ActionExecutor to apply a pervasive check to all derived classes, define an increase in scope, and can be isolated from this one.

          Show
          Mona Chitnis added a comment - I agree that the recent changes in ActionExecutor to apply a pervasive check to all derived classes, define an increase in scope, and can be isolated from this one.
          Hide
          Jeremy Hanna added a comment -

          If we are still blocked on this, I can work on getting the more complete patch going.

          Show
          Jeremy Hanna added a comment - If we are still blocked on this, I can work on getting the more complete patch going.
          Hide
          Mona Chitnis added a comment -

          Jeremy,

          Thanks for bringing it up. I was blocked on the patch before since we were reviewing a partial patch in time for 3.2. Now the complete patch with some refactoring and path check in worflow/coordinator/bundle submission will be out for review. I had already started working on it through Reviewboard so I can continue and upload a patch here shortly.

          Mona

          Show
          Mona Chitnis added a comment - Jeremy, Thanks for bringing it up. I was blocked on the patch before since we were reviewing a partial patch in time for 3.2. Now the complete patch with some refactoring and path check in worflow/coordinator/bundle submission will be out for review. I had already started working on it through Reviewboard so I can continue and upload a patch here shortly. Mona
          Hide
          Jeremy Hanna added a comment -

          Mona,

          Thanks for letting me know. I didn't realize you were working on the full solution.

          Thanks!

          Jeremy

          Show
          Jeremy Hanna added a comment - Mona, Thanks for letting me know. I didn't realize you were working on the full solution. Thanks! Jeremy
          Hide
          Mona Chitnis added a comment -

          AE classes unit tests pass.

          Show
          Mona Chitnis added a comment - AE classes unit tests pass.
          Hide
          Alejandro Abdelnur added a comment -

          Mona, looking at the latest patch in reviewboard (which I assume is not the latest greatest you are working on) I'm wondering if wouldn't be easier to just remove all scheme checks and multiclustersupport settings (which i'm not jut I get correctly); and just make the HadoopAccessorService to enforce that the scheme of the URI of the FS requested is whitelisted in an oozie configuration property.

          A good way of testing that we don't have any regressions would be to configure a viewfs:// filesystem and run all the examples thru it where everything uses viewfs://.

          Also, we should check that things work with HDFS HA introduced in Hadoop2

          Show
          Alejandro Abdelnur added a comment - Mona, looking at the latest patch in reviewboard (which I assume is not the latest greatest you are working on) I'm wondering if wouldn't be easier to just remove all scheme checks and multiclustersupport settings (which i'm not jut I get correctly); and just make the HadoopAccessorService to enforce that the scheme of the URI of the FS requested is whitelisted in an oozie configuration property. A good way of testing that we don't have any regressions would be to configure a viewfs:// filesystem and run all the examples thru it where everything uses viewfs://. Also, we should check that things work with HDFS HA introduced in Hadoop2
          Hide
          Mona Chitnis added a comment -

          The latest updated patch is the one i've attached to this JIRA. (removed the extra multiclustersupport=true/false setting since the oozie property for list of supported schemes "oozie.filesystems.supported" specifies the whitelist). I agree that the change in HadoopAccessorService would be good. That's what I started with before but that opened up chances of regression in a lot of places e.g. checking coordinator input dataset dependencies. So we are also enforcing datasets to lie in supported schemes only, right?

          In the meanwhile, I can look into the viewfs setup.

          Show
          Mona Chitnis added a comment - The latest updated patch is the one i've attached to this JIRA. (removed the extra multiclustersupport=true/false setting since the oozie property for list of supported schemes "oozie.filesystems.supported" specifies the whitelist). I agree that the change in HadoopAccessorService would be good. That's what I started with before but that opened up chances of regression in a lot of places e.g. checking coordinator input dataset dependencies. So we are also enforcing datasets to lie in supported schemes only, right? In the meanwhile, I can look into the viewfs setup.
          Hide
          Mona Chitnis added a comment -

          address review comments - uri scheme check in HadoopAccessorService. Unit tests pass

          Show
          Mona Chitnis added a comment - address review comments - uri scheme check in HadoopAccessorService. Unit tests pass
          Hide
          Alejandro Abdelnur added a comment -

          Thanks Mona, following feedback on the latest patch.

          FSActionExecutor

          I'm a bit puzzled by this line:

          fsConf.set(has.SUPPORTED_FILESYSTEMS, System.getProperty(has.SUPPORTED_FILESYSTEMS));

          All this method should go away and simply do a Services.get().get(HadoopAcessorService.class).checkSupportFileSystem(URI) inline.

          HadoopAccessorService.checkSupporteFileSystem()

          Instead expecting a fileScheme, we should expect a URI and check the scheme of it.

          we should not pass a configuration object, instead we should used the oozie configuration to check for the supported filesystems.

          we should load the supported filesystems in a set at init time, not to do this logic over an over.

          there is are return deep in an IF, instead we should set 'isSupported = true'. The 'if(supportedSchemes != null) should be replaced with an ELSE.

          Instead returning true/false and delegating to the caller to fail, the checkSupportedFileSystem() method should throw a HadoopAccessorException.

          Show
          Alejandro Abdelnur added a comment - Thanks Mona, following feedback on the latest patch. FSActionExecutor I'm a bit puzzled by this line: fsConf.set(has.SUPPORTED_FILESYSTEMS, System.getProperty(has.SUPPORTED_FILESYSTEMS)); All this method should go away and simply do a Services.get().get(HadoopAcessorService.class).checkSupportFileSystem(URI) inline. HadoopAccessorService.checkSupporteFileSystem() Instead expecting a fileScheme, we should expect a URI and check the scheme of it. we should not pass a configuration object, instead we should used the oozie configuration to check for the supported filesystems. we should load the supported filesystems in a set at init time, not to do this logic over an over. there is are return deep in an IF, instead we should set 'isSupported = true'. The 'if(supportedSchemes != null) should be replaced with an ELSE. Instead returning true/false and delegating to the caller to fail, the checkSupportedFileSystem() method should throw a HadoopAccessorException.
          Hide
          Mona Chitnis added a comment -

          Put the block to populate supportedSchemes set from conf, at HAS init time. Since the checkSupportedFileSystem now throws HASException, the flag 'isSupported' is unnecessary and removed. Cleaner calling of this function since configuration object is not passed as arg.

          Show
          Mona Chitnis added a comment - Put the block to populate supportedSchemes set from conf, at HAS init time. Since the checkSupportedFileSystem now throws HASException, the flag 'isSupported' is unnecessary and removed. Cleaner calling of this function since configuration object is not passed as arg.
          Hide
          Mona Chitnis added a comment -

          Above comment for patch v3

          Show
          Mona Chitnis added a comment - Above comment for patch v3
          Hide
          Mona Chitnis added a comment -

          Fix in LauncherMapper where filesystem for particular URI scheme and authority is obtained.

          Show
          Mona Chitnis added a comment - Fix in LauncherMapper where filesystem for particular URI scheme and authority is obtained.
          Hide
          Mona Chitnis added a comment -

          @Jeremy,

          Since you had a viewfs setup ready before, will it be feasible for you to test this patch against it? Please let me know.

          Thanks,
          Mona

          Show
          Mona Chitnis added a comment - @Jeremy, Since you had a viewfs setup ready before, will it be feasible for you to test this patch against it? Please let me know. Thanks, Mona
          Hide
          Jeremy Hanna added a comment -

          My setup was an alternative filesystem that I could try again. I'm in the middle of something now, but could do that some time next week if that works.

          Show
          Jeremy Hanna added a comment - My setup was an alternative filesystem that I could try again. I'm in the middle of something now, but could do that some time next week if that works.
          Hide
          Jeremy Hanna added a comment -

          Mona - I'm trying to apply the v4 patch but am unable to. I tried against 3.2-branch, release-3.2.0-incubating tag, and trunk. It looks like it's against trunk though. Do you mind rebasing on trunk or the preferred branch? Sorry about taking a little while to test it out.

          Show
          Jeremy Hanna added a comment - Mona - I'm trying to apply the v4 patch but am unable to. I tried against 3.2-branch, release-3.2.0-incubating tag, and trunk. It looks like it's against trunk though. Do you mind rebasing on trunk or the preferred branch? Sorry about taking a little while to test it out.
          Hide
          Mona Chitnis added a comment -

          A couple of files got updated on trunk since my last patch. Attaching the patch after rebasing for TRUNK. Sorry about that Jeremy.

          Mona

          Show
          Mona Chitnis added a comment - A couple of files got updated on trunk since my last patch. Attaching the patch after rebasing for TRUNK. Sorry about that Jeremy. Mona
          Hide
          Jeremy Hanna added a comment -

          It was my fault for not testing it soon enough. It applies cleanly now. Thank you!

          Show
          Jeremy Hanna added a comment - It was my fault for not testing it soon enough. It applies cleanly now. Thank you!
          Hide
          Jeremy Hanna added a comment -

          Hi Mona,

          I'm getting an exception for "No filesystem for scheme" when I had specified the filesystem defined in the oozie-site.xml. Any ideas as to why? Is it some check that was not configured?

          Here is my config property in the oozie-site.xml:
          <property>
          <name>oozie.filesystems.supported</name>
          <value>hdfs,cfs</value>
          <description>
          Enlist the different filesystems supported for federation.
          </description>
          </property>

          Here is the stack trace:
          2012-07-22 01:33:06,169 WARN V1JobsServlet:542 - USER[oozie] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST http://localhost:11000/oozie/v1/jobs?action=start] error[E0902], E0902: Exception occured: [java.io.IOException: No FileSystem for scheme: cfs]
          org.apache.oozie.servlet.XServletException: E0902: Exception occured: [java.io.IOException: No FileSystem for scheme: cfs]
          at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:207)
          at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:92)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
          at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:285)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
          at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:126)
          at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:382)
          at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
          at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:67)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
          at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
          at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
          at java.lang.Thread.run(Thread.java:662)
          Caused by: org.apache.oozie.service.AuthorizationException: E0902: Exception occured: [java.io.IOException: No FileSystem for scheme: cfs]
          at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:401)
          at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:196)
          ... 24 more
          Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [java.io.IOException: No FileSystem for scheme: cfs]
          at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:410)
          at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:366)
          ... 25 more
          Caused by: java.io.IOException: No FileSystem for scheme: cfs
          at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1383)
          at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
          at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
          at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
          at org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:402)
          at org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:400)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:396)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
          at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:400)
          ... 26 more

          Show
          Jeremy Hanna added a comment - Hi Mona, I'm getting an exception for "No filesystem for scheme" when I had specified the filesystem defined in the oozie-site.xml. Any ideas as to why? Is it some check that was not configured? Here is my config property in the oozie-site.xml: <property> <name>oozie.filesystems.supported</name> <value>hdfs,cfs</value> <description> Enlist the different filesystems supported for federation. </description> </property> Here is the stack trace: 2012-07-22 01:33:06,169 WARN V1JobsServlet:542 - USER [oozie] GROUP [-] TOKEN [-] APP [-] JOB [-] ACTION [-] URL [POST http://localhost:11000/oozie/v1/jobs?action=start] error [E0902] , E0902: Exception occured: [java.io.IOException: No FileSystem for scheme: cfs] org.apache.oozie.servlet.XServletException: E0902: Exception occured: [java.io.IOException: No FileSystem for scheme: cfs] at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:207) at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:92) at javax.servlet.http.HttpServlet.service(HttpServlet.java:637) at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:285) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:126) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:382) at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:67) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.oozie.service.AuthorizationException: E0902: Exception occured: [java.io.IOException: No FileSystem for scheme: cfs] at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:401) at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:196) ... 24 more Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [java.io.IOException: No FileSystem for scheme: cfs] at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:410) at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:366) ... 25 more Caused by: java.io.IOException: No FileSystem for scheme: cfs at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1383) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:402) at org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:400) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:400) ... 26 more
          Hide
          Jeremy Hanna added a comment -

          Hmm - I think I found out why - I need to add the core-site.xml that I'm using as well as the jar with the class it needs into the oozie-server/lib directory for it to recognize it. It didn't have to do with oozie in this case.

          Show
          Jeremy Hanna added a comment - Hmm - I think I found out why - I need to add the core-site.xml that I'm using as well as the jar with the class it needs into the oozie-server/lib directory for it to recognize it. It didn't have to do with oozie in this case.
          Hide
          Jeremy Hanna added a comment -

          Mona: the patch appears to work fine - gets past submission errors for me. Thanks!

          Show
          Jeremy Hanna added a comment - Mona: the patch appears to work fine - gets past submission errors for me. Thanks!
          Hide
          Mona Chitnis added a comment -

          Thanks Jeremy for the confirmation!

          Alejandro, can this be pushed since its tested with the alternate fs?

          Show
          Mona Chitnis added a comment - Thanks Jeremy for the confirmation! Alejandro, can this be pushed since its tested with the alternate fs?
          Hide
          Mona Chitnis added a comment -

          v5: Made small changes to setup config of 2 unit tests. Now all tests pass

          Show
          Mona Chitnis added a comment - v5: Made small changes to setup config of 2 unit tests. Now all tests pass
          Hide
          Jeremy Hanna added a comment -

          Is there anything holding this up from being added to the 3.2-branch and trunk?

          Thanks!

          Show
          Jeremy Hanna added a comment - Is there anything holding this up from being added to the 3.2-branch and trunk? Thanks!
          Hide
          Alejandro Abdelnur added a comment -

          i'll review it TUE AUG07. THX

          Show
          Alejandro Abdelnur added a comment - i'll review it TUE AUG07. THX
          Hide
          Alejandro Abdelnur added a comment -

          Mona, the patch fails to apply to trunk, would you please rebase it?

          Show
          Alejandro Abdelnur added a comment - Mona, the patch fails to apply to trunk, would you please rebase it?
          Hide
          Mona Chitnis added a comment -

          Hi Tucu,

          v5 patch rebased to apply cleanly to trunk

          Thanks,

          Show
          Mona Chitnis added a comment - Hi Tucu, v5 patch rebased to apply cleanly to trunk Thanks,
          Hide
          Mona Chitnis added a comment -

          Latest updated patch on Reviewboard

          • rebased to trunk
          • addressed tucu's comments
          Show
          Mona Chitnis added a comment - Latest updated patch on Reviewboard rebased to trunk addressed tucu's comments
          Hide
          Mona Chitnis added a comment -

          final comments addressed

          • Wildcard "*" support for allowing all schemes
          • 'oozie.libpath' cannot be relative
          Show
          Mona Chitnis added a comment - final comments addressed Wildcard "*" support for allowing all schemes 'oozie.libpath' cannot be relative
          Hide
          Mona Chitnis added a comment -

          Final patch +1'ed on ReviewBoard

          Show
          Mona Chitnis added a comment - Final patch +1'ed on ReviewBoard
          Hide
          Hadoop QA added a comment -

          Testing JIRA OOZIE-477

          ----------------------------

          +1 PATCH_APPLIES
          CLEAN cleaned target directories
          +1 RAT
          +1 the patch does not seem to introduce new RAT warnings
          -1 COMPILE
          -1 HEAD does not compile
          -1 patch does not compile
          +1 the patch does not seem to introduce new javac warnings
          +1 TESTS
          Tests run: 0
          Tests failures: 0
          Tests errors: 0

          ----------------------------

          The full output of the test-patch run is available at

          https://builds.apache.org/job/oozie-trunk-precommit-build/29/

          Show
          Hadoop QA added a comment - Testing JIRA OOZIE-477 ---------------------------- +1 PATCH_APPLIES CLEAN cleaned target directories +1 RAT +1 the patch does not seem to introduce new RAT warnings -1 COMPILE -1 HEAD does not compile -1 patch does not compile +1 the patch does not seem to introduce new javac warnings +1 TESTS Tests run: 0 Tests failures: 0 Tests errors: 0 ---------------------------- The full output of the test-patch run is available at https://builds.apache.org/job/oozie-trunk-precommit-build/29/
          Hide
          Hadoop QA added a comment -

          Testing JIRA OOZIE-477

          ----------------------------

          +1 PATCH_APPLIES
          CLEAN cleaned target directories
          +1 RAT
          +1 the patch does not seem to introduce new RAT warnings
          +1 COMPILE
          +1 HEAD compiles
          +1 patch compiles
          +1 the patch does not seem to introduce new javac warnings
          -1 TESTS - the patch failed the following testcases:

          org.apache.oozie.service.TestStatusTransitService
          org.apache.oozie.command.bundle.TestBundleRerunXCommand

          Tests run: 874
          Tests failures: 3
          Tests errors: 0

          ----------------------------

          The full output of the test-patch run is available at

          https://builds.apache.org/job/oozie-trunk-precommit-build/30/

          Show
          Hadoop QA added a comment - Testing JIRA OOZIE-477 ---------------------------- +1 PATCH_APPLIES CLEAN cleaned target directories +1 RAT +1 the patch does not seem to introduce new RAT warnings +1 COMPILE +1 HEAD compiles +1 patch compiles +1 the patch does not seem to introduce new javac warnings -1 TESTS - the patch failed the following testcases: org.apache.oozie.service.TestStatusTransitService org.apache.oozie.command.bundle.TestBundleRerunXCommand Tests run: 874 Tests failures: 3 Tests errors: 0 ---------------------------- The full output of the test-patch run is available at https://builds.apache.org/job/oozie-trunk-precommit-build/30/
          Hide
          Alejandro Abdelnur added a comment -

          Thanks Mona, Virag & Jeremy. Committed to trunk.

          Show
          Alejandro Abdelnur added a comment - Thanks Mona, Virag & Jeremy. Committed to trunk.
          Hide
          Alejandro Abdelnur added a comment -

          Closing issue as it has been released as part of 3.3.0 release

          Show
          Alejandro Abdelnur added a comment - Closing issue as it has been released as part of 3.3.0 release

            People

            • Assignee:
              Mona Chitnis
              Reporter:
              Mayank Bansal
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development