Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.3
    • Fix Version/s: 0.4
    • Component/s: oozie
    • Labels:
      None

      Description

      Hive/HCatalog based data availability triggers are available in Oozie 4.x and Hive integration with Falcon can take advantage of this. Hence the need to upgrade. There is an RC3: http://people.apache.org/~mona/oozie-4.0.0-rc3/ and may wait until we have a final release.

      1. FALCON-103.patch
        81 kB
        Venkatesh Seetharam
      2. FALCON-103-r1.patch
        84 kB
        Venkatesh Seetharam
      3. FALCON-103-r2.patch
        112 kB
        Venkatesh Seetharam

        Issue Links

          Activity

          Hide
          Venkatesh Seetharam added a comment -

          Phew!!! This took more effort than I anticipated. Oozie 4.x has changed some client API, got into couple of jar conflict hell, log4j and hcat. I haven't tested hcat and oozie integration but will add as part of FALCON-94.

          All unit and integration tests pass.

          Show
          Venkatesh Seetharam added a comment - Phew!!! This took more effort than I anticipated. Oozie 4.x has changed some client API, got into couple of jar conflict hell, log4j and hcat. I haven't tested hcat and oozie integration but will add as part of FALCON-94 . All unit and integration tests pass.
          Hide
          Srikanth Sundarrajan added a comment -

          We should look to create a patched oozie version as 4.0.0-falcon or some such thing. I think currently it is still 3.2.2 in this patch. Also src/bin/package.sh under root also needs change.

          Looks good otherwise.

          Show
          Srikanth Sundarrajan added a comment - We should look to create a patched oozie version as 4.0.0-falcon or some such thing. I think currently it is still 3.2.2 in this patch. Also src/bin/package.sh under root also needs change. Looks good otherwise.
          Hide
          Srikanth Sundarrajan added a comment -

          Tried applying this directly and also post FALCON-102, FALCON-96 (both patches), still doesn't apply. can you check.

          sriksun:falcon-for-commit sriksun$ curl "https://issues.apache.org/jira/secure/attachment/12601771/FALCON-103.patch" | git apply --check -v
            % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                           Dload  Upload   Total   Spent    Left  Speed
          100 82842  100 82842    0     0   3382      0  0:00:24  0:00:24 --:--:--  7689
          Checking patch build-tools/pom.xml...
          Checking patch build-tools/src/bin/OOZIE-674.sh...
          Checking patch build-tools/src/bin/oozie-patch-pkg.sh...
          Checking patch build-tools/src/patch/oozie-4.0.0.patch...
          Checking patch client/src/main/java/org/apache/falcon/entity/v0/Frequency.java...
          Checking patch client/src/test/java/org/apache/falcon/entity/v0/TestFrequency.java...
          Checking patch common/src/main/java/org/apache/falcon/entity/EntityUtil.java...
          Checking patch common/src/main/resources/log4j.xml...
          Checking patch hadoop-webapp/pom.xml...
          Checking patch hadoop-webapp/src/main/resources/log4j.xml...
          Checking patch messaging/src/main/resources/log4j.xml...
          Checking patch oozie-3.2.0-incubating-el.patch...
          Checking patch oozie-bundle-el-extension.patch...
          error: while searching for:
          diff --git webapp/pom.xml webapp/pom.xml
          index 38ec438..c3bc9b8 100644
          --- webapp/pom.xml
          +++ webapp/pom.xml
          @@ -33,6 +33,13 @@
           
               <dependencies>
                   <dependency>
          +            <groupId>org.apache.falcon</groupId>
          +            <artifactId>falcon-oozie-el-extension</artifactId>
          +            <version>0.4-SNAPSHOT</version>
          +            <scope>compile</scope>
          +        </dependency>
          +
          +        <dependency>
                       <groupId>org.apache.oozie</groupId>
                       <artifactId>oozie-core</artifactId>
                       <scope>compile</scope>
          
          error: patch failed: oozie-bundle-el-extension.patch:1
          error: oozie-bundle-el-extension.patch: patch does not apply
          Checking patch oozie-el-extensions/pom.xml...
          Checking patch oozie-el-extensions/src/main/java/org/apache/oozie/extensions/OozieELExtensions.java...
          Checking patch oozie-el-extensions/src/test/java/org/apache/oozie/extensions/TestOozieELExtensions.java...
          Checking patch oozie/src/main/java/org/apache/falcon/workflow/engine/NullCoordJob.java...
          Checking patch oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java...
          Checking patch pom.xml...
          Hunk #2 succeeded at 183 (offset 1 lines).
          Hunk #3 succeeded at 257 (offset 1 lines).
          Hunk #4 succeeded at 581 (offset 1 lines).
          Hunk #5 succeeded at 618 (offset 1 lines).
          Hunk #6 succeeded at 645 (offset 1 lines).
          Hunk #7 succeeded at 778 (offset 1 lines).
          Checking patch prism/src/main/java/org/apache/falcon/service/SLAMonitoringService.java...
          Checking patch prism/src/main/resources/log4j.xml...
          Checking patch webapp/src/main/resources/log4j.xml...
          Checking patch webapp/src/main/webapp/index.html...
          
          Show
          Srikanth Sundarrajan added a comment - Tried applying this directly and also post FALCON-102 , FALCON-96 (both patches), still doesn't apply. can you check. sriksun:falcon- for -commit sriksun$ curl "https: //issues.apache.org/jira/secure/attachment/12601771/FALCON-103.patch" | git apply --check -v % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 82842 100 82842 0 0 3382 0 0:00:24 0:00:24 --:--:-- 7689 Checking patch build-tools/pom.xml... Checking patch build-tools/src/bin/OOZIE-674.sh... Checking patch build-tools/src/bin/oozie-patch-pkg.sh... Checking patch build-tools/src/patch/oozie-4.0.0.patch... Checking patch client/src/main/java/org/apache/falcon/entity/v0/Frequency.java... Checking patch client/src/test/java/org/apache/falcon/entity/v0/TestFrequency.java... Checking patch common/src/main/java/org/apache/falcon/entity/EntityUtil.java... Checking patch common/src/main/resources/log4j.xml... Checking patch hadoop-webapp/pom.xml... Checking patch hadoop-webapp/src/main/resources/log4j.xml... Checking patch messaging/src/main/resources/log4j.xml... Checking patch oozie-3.2.0-incubating-el.patch... Checking patch oozie-bundle-el-extension.patch... error: while searching for : diff --git webapp/pom.xml webapp/pom.xml index 38ec438..c3bc9b8 100644 --- webapp/pom.xml +++ webapp/pom.xml @@ -33,6 +33,13 @@ <dependencies> <dependency> + <groupId>org.apache.falcon</groupId> + <artifactId>falcon-oozie-el-extension</artifactId> + <version>0.4-SNAPSHOT</version> + <scope>compile</scope> + </dependency> + + <dependency> <groupId>org.apache.oozie</groupId> <artifactId>oozie-core</artifactId> <scope>compile</scope> error: patch failed: oozie-bundle-el-extension.patch:1 error: oozie-bundle-el-extension.patch: patch does not apply Checking patch oozie-el-extensions/pom.xml... Checking patch oozie-el-extensions/src/main/java/org/apache/oozie/extensions/OozieELExtensions.java... Checking patch oozie-el-extensions/src/test/java/org/apache/oozie/extensions/TestOozieELExtensions.java... Checking patch oozie/src/main/java/org/apache/falcon/workflow/engine/NullCoordJob.java... Checking patch oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java... Checking patch pom.xml... Hunk #2 succeeded at 183 (offset 1 lines). Hunk #3 succeeded at 257 (offset 1 lines). Hunk #4 succeeded at 581 (offset 1 lines). Hunk #5 succeeded at 618 (offset 1 lines). Hunk #6 succeeded at 645 (offset 1 lines). Hunk #7 succeeded at 778 (offset 1 lines). Checking patch prism/src/main/java/org/apache/falcon/service/SLAMonitoringService.java... Checking patch prism/src/main/resources/log4j.xml... Checking patch webapp/src/main/resources/log4j.xml... Checking patch webapp/src/main/webapp/index.html...
          Hide
          Venkatesh Seetharam added a comment -

          This should be an independent patch and does not depend on anything else. Wonder why. Will see.

          We should look to create a patched oozie version as 4.0.0-falcon or some such thing. I think currently it is still 3.2.2 in this patch.

          I have changed it. Pls check again in target dir and your maven repo.

          rm -rf oozie-*
          curl -v "http://www.apache.org/dist/oozie/4.0.0/oozie-4.0.0.tar.gz" -o oozie-4.0.0.tgz
          tar -xzvf oozie-4.0.0.tgz
          cd oozie-4.0.0
          pwd
          patch -p0 < ../../build-tools/src/patch/oozie-4.0.0.patch
          

          Also src/bin/package.sh under root also needs change.

          I missed that. Thanks for catching this. But why yet another one? This will not be necessary and cant code from /src/bin/oozie-patch-pkg.sh be reused?

          Show
          Venkatesh Seetharam added a comment - This should be an independent patch and does not depend on anything else. Wonder why. Will see. We should look to create a patched oozie version as 4.0.0-falcon or some such thing. I think currently it is still 3.2.2 in this patch. I have changed it. Pls check again in target dir and your maven repo. rm -rf oozie-* curl -v "http: //www.apache.org/dist/oozie/4.0.0/oozie-4.0.0.tar.gz" -o oozie-4.0.0.tgz tar -xzvf oozie-4.0.0.tgz cd oozie-4.0.0 pwd patch -p0 < ../../build-tools/src/patch/oozie-4.0.0.patch Also src/bin/package.sh under root also needs change. I missed that. Thanks for catching this. But why yet another one? This will not be necessary and cant code from /src/bin/oozie-patch-pkg.sh be reused?
          Hide
          Venkatesh Seetharam added a comment -

          Attached new patch incorporating review comments.

          Show
          Venkatesh Seetharam added a comment - Attached new patch incorporating review comments.
          Hide
          Venkatesh Seetharam added a comment -

          Rebased on trunk and patch revised for resolving conflicts. Tested that it applies cleanly and all tests pass.

          Show
          Venkatesh Seetharam added a comment - Rebased on trunk and patch revised for resolving conflicts. Tested that it applies cleanly and all tests pass.
          Hide
          Shwetha G S added a comment -

          Can you make the oozie version configurable which is passed to the script using the variable defined in main pom. The version shouldn't be hard coded in scripts/patch so that upgrading to new version requires just change of version in main pom.

          Show
          Shwetha G S added a comment - Can you make the oozie version configurable which is passed to the script using the variable defined in main pom. The version shouldn't be hard coded in scripts/patch so that upgrading to new version requires just change of version in main pom.
          Hide
          Venkatesh Seetharam added a comment -

          Good suggestion Shwetha. But I'll let this slip into another jira. Thanks!

          Show
          Venkatesh Seetharam added a comment - Good suggestion Shwetha. But I'll let this slip into another jira. Thanks!
          Hide
          Srikanth Sundarrajan added a comment -

          FALCON-66 tries to address upgrading to specific version of oozie in a generic way. This one should perhaps reconcile once those changes are committed.

          Show
          Srikanth Sundarrajan added a comment - FALCON-66 tries to address upgrading to specific version of oozie in a generic way. This one should perhaps reconcile once those changes are committed.
          Hide
          Venkatesh Seetharam added a comment -

          This one should perhaps reconcile once those changes are committed.

          I understand if this is on trunk but since this is on a feature branch, this can be committed and reconciled from time to time or during merge. Else making progress on other issues would be very hard. Makes sense?

          Show
          Venkatesh Seetharam added a comment - This one should perhaps reconcile once those changes are committed. I understand if this is on trunk but since this is on a feature branch, this can be committed and reconciled from time to time or during merge. Else making progress on other issues would be very hard. Makes sense?
          Hide
          Srikanth Sundarrajan added a comment -

          Yes. This can be committed to FALCON-85 branch, but might have to be fixed post FALCON-66 before FALCON-85 branch is merged to mainline.

          Show
          Srikanth Sundarrajan added a comment - Yes. This can be committed to FALCON-85 branch, but might have to be fixed post FALCON-66 before FALCON-85 branch is merged to mainline.
          Hide
          Srikanth Sundarrajan added a comment -

          Patch doesn't apply. Can you please check this

          sriksun:falcon-for-commit sriksun$ curl "https://issues.apache.org/jira/secure/attachment/12602225/FALCON-103-r2.patch" | git apply --check -v
          Checking patch Installation-steps.txt...
          Checking patch build-tools/pom.xml...
          Checking patch build-tools/src/bin/OOZIE-674.sh...
          Checking patch build-tools/src/bin/oozie-patch-pkg.sh...
          Checking patch build-tools/src/patch/oozie-4.0.0-falcon.patch...
          Checking patch build-tools/src/patch/oozie-bundle-el-extension.patch...
          Checking patch client/src/main/java/org/apache/falcon/entity/v0/Frequency.java...
          Checking patch client/src/test/java/org/apache/falcon/entity/v0/TestFrequency.java...
          Checking patch common/src/main/java/org/apache/falcon/entity/EntityUtil.java...
          Checking patch common/src/main/resources/log4j.xml...
          Checking patch docs/src/site/twiki/InstallationSteps.twiki...
          Checking patch hadoop-webapp/pom.xml...
          error: while searching for:
                      <groupId>org.apache.activemq</groupId>
                      <artifactId>activemq-core</artifactId>
                  </dependency>
              </dependencies>
          
              <build>
          
          error: patch failed: hadoop-webapp/pom.xml:91
          error: hadoop-webapp/pom.xml: patch does not apply
          Checking patch hadoop-webapp/src/main/resources/log4j.xml...
          Checking patch messaging/src/main/resources/log4j.xml...
          Checking patch oozie-3.2.0-incubating-el.patch...
          Checking patch oozie-bundle-el-extension.patch...
          Checking patch oozie-el-extensions/src/main/java/org/apache/oozie/extensions/OozieELExtensions.java...
          Checking patch oozie-el-extensions/src/test/java/org/apache/oozie/extensions/TestOozieELExtensions.java...
          Checking patch oozie/src/main/java/org/apache/falcon/workflow/engine/NullCoordJob.java...
          Checking patch oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java...
          Checking patch pom.xml...
          error: while searching for:
          
                  <hadoop.profile>1</hadoop.profile>
                  <hadoop.version>1.1.2</hadoop.version>
                  <slf4j.version>1.2</slf4j.version>
                  <oozie.version>3.2.2</oozie.version>
                  <activemq.version>5.4.3</activemq.version>
                  <hadoop-distcp.version>0.3</hadoop-distcp.version>
                  <hive.version>0.11.0</hive.version>
          
          error: patch failed: pom.xml:98
          error: pom.xml: patch does not apply
          Checking patch prism/src/main/java/org/apache/falcon/service/SLAMonitoringService.java...
          Checking patch prism/src/main/resources/log4j.xml...
          Checking patch src/bin/package.sh...
          Checking patch webapp/src/main/resources/log4j.xml...
          Checking patch webapp/src/main/webapp/index.html...
          
          Show
          Srikanth Sundarrajan added a comment - Patch doesn't apply. Can you please check this sriksun:falcon- for -commit sriksun$ curl "https: //issues.apache.org/jira/secure/attachment/12602225/FALCON-103-r2.patch" | git apply --check -v Checking patch Installation-steps.txt... Checking patch build-tools/pom.xml... Checking patch build-tools/src/bin/OOZIE-674.sh... Checking patch build-tools/src/bin/oozie-patch-pkg.sh... Checking patch build-tools/src/patch/oozie-4.0.0-falcon.patch... Checking patch build-tools/src/patch/oozie-bundle-el-extension.patch... Checking patch client/src/main/java/org/apache/falcon/entity/v0/Frequency.java... Checking patch client/src/test/java/org/apache/falcon/entity/v0/TestFrequency.java... Checking patch common/src/main/java/org/apache/falcon/entity/EntityUtil.java... Checking patch common/src/main/resources/log4j.xml... Checking patch docs/src/site/twiki/InstallationSteps.twiki... Checking patch hadoop-webapp/pom.xml... error: while searching for : <groupId>org.apache.activemq</groupId> <artifactId>activemq-core</artifactId> </dependency> </dependencies> <build> error: patch failed: hadoop-webapp/pom.xml:91 error: hadoop-webapp/pom.xml: patch does not apply Checking patch hadoop-webapp/src/main/resources/log4j.xml... Checking patch messaging/src/main/resources/log4j.xml... Checking patch oozie-3.2.0-incubating-el.patch... Checking patch oozie-bundle-el-extension.patch... Checking patch oozie-el-extensions/src/main/java/org/apache/oozie/extensions/OozieELExtensions.java... Checking patch oozie-el-extensions/src/test/java/org/apache/oozie/extensions/TestOozieELExtensions.java... Checking patch oozie/src/main/java/org/apache/falcon/workflow/engine/NullCoordJob.java... Checking patch oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java... Checking patch pom.xml... error: while searching for : <hadoop.profile>1</hadoop.profile> <hadoop.version>1.1.2</hadoop.version> <slf4j.version>1.2</slf4j.version> <oozie.version>3.2.2</oozie.version> <activemq.version>5.4.3</activemq.version> <hadoop-distcp.version>0.3</hadoop-distcp.version> <hive.version>0.11.0</hive.version> error: patch failed: pom.xml:98 error: pom.xml: patch does not apply Checking patch prism/src/main/java/org/apache/falcon/service/SLAMonitoringService.java... Checking patch prism/src/main/resources/log4j.xml... Checking patch src/bin/ package .sh... Checking patch webapp/src/main/resources/log4j.xml... Checking patch webapp/src/main/webapp/index.html...
          Hide
          Srikanth Sundarrajan added a comment -

          Patch generally looks good. Please commit to FALCON-85 post re-base once all the tests pass.

          Show
          Srikanth Sundarrajan added a comment - Patch generally looks good. Please commit to FALCON-85 post re-base once all the tests pass.
          Hide
          Venkatesh Seetharam added a comment - - edited

          Thanks Srikanth Sundarrajan. All unit and integration tests pass. Committed this to FALCON-85.

          PS: I'd like to add that FALCON-87 is reverted in FALCON-85 branch which I had to remove before proceeding with this.

          Show
          Venkatesh Seetharam added a comment - - edited Thanks Srikanth Sundarrajan . All unit and integration tests pass. Committed this to FALCON-85 . PS: I'd like to add that FALCON-87 is reverted in FALCON-85 branch which I had to remove before proceeding with this.
          Hide
          Srikanth Sundarrajan added a comment -

          PS: I'd like to add that FALCON-87 is reverted in FALCON-85 branch which I had to remove before proceeding with this.

          From https://git-wip-us.apache.org/repos/asf?p=incubator-falcon.git;a=shortlog;h=refs/heads/FALCON-85, it doesn't seem like any changes were reverted, you might accidentally be checking the mainline.

          Show
          Srikanth Sundarrajan added a comment - PS: I'd like to add that FALCON-87 is reverted in FALCON-85 branch which I had to remove before proceeding with this. From https://git-wip-us.apache.org/repos/asf?p=incubator-falcon.git;a=shortlog;h=refs/heads/FALCON-85 , it doesn't seem like any changes were reverted, you might accidentally be checking the mainline.

            People

            • Assignee:
              Venkatesh Seetharam
              Reporter:
              Venkatesh Seetharam
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development