Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: None
    • Labels:
      None

      Description

      If we upgrade Oozie to 4.2.0, It would be able to leverage existing components:

      • Actions for Hiveserver2 and Spark
      • Workflow editor on HUE
      1. BIGTOP-1915-1.patch
        9 kB
        Mike Grimes
      2. BIGTOP-1915-2.patch
        21 kB
        Mike Grimes
      3. BIGTOP-1915-3.patch
        21 kB
        Mike Grimes

        Issue Links

          Activity

          Hide
          warwithin YoungWoo Kim added a comment -

          Needed OOZIE-2305 to make build work with Hive 1.0+ and Oozie 4.2+

          Show
          warwithin YoungWoo Kim added a comment - Needed OOZIE-2305 to make build work with Hive 1.0+ and Oozie 4.2+
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user youngwookim opened a pull request:

          https://github.com/apache/bigtop/pull/28

          BIGTOP-1915. Upgrade Oozie to 4.2.0

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/youngwookim/bigtop BIGTOP-1915

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/28.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #28


          commit 59df96a9ffc6063743b90331d3202ce8e3ffe0a7
          Author: YoungWoo Kim <ywkim@apache.org>
          Date: 2015-07-01T07:22:53Z

          BIGTOP-1915. Upgrade Oozie to 4.2.0


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user youngwookim opened a pull request: https://github.com/apache/bigtop/pull/28 BIGTOP-1915 . Upgrade Oozie to 4.2.0 You can merge this pull request into a Git repository by running: $ git pull https://github.com/youngwookim/bigtop BIGTOP-1915 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/28.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #28 commit 59df96a9ffc6063743b90331d3202ce8e3ffe0a7 Author: YoungWoo Kim <ywkim@apache.org> Date: 2015-07-01T07:22:53Z BIGTOP-1915 . Upgrade Oozie to 4.2.0
          Hide
          grimesmi Mike Grimes added a comment - - edited

          +1

          Although, I did have to make some minor changes to the patch to get it to apply correctly. I was able to get the oozie-4.2.0 RPMs to build successfully with this. YoungWoo Kim Let me know if you'd like to see the patch changes, or would like me to upload the new one here.

          Show
          grimesmi Mike Grimes added a comment - - edited +1 Although, I did have to make some minor changes to the patch to get it to apply correctly. I was able to get the oozie-4.2.0 RPMs to build successfully with this. YoungWoo Kim Let me know if you'd like to see the patch changes, or would like me to upload the new one here.
          Hide
          warwithin YoungWoo Kim added a comment -

          Mike Grimes Thank you for looking into the patch. Good to know that patch works on your end Feel free to revise it! It would be great if you upload the revised one.

          Show
          warwithin YoungWoo Kim added a comment - Mike Grimes Thank you for looking into the patch. Good to know that patch works on your end Feel free to revise it! It would be great if you upload the revised one.
          Hide
          grimesmi Mike Grimes added a comment -

          Only includes update to oozie 4.2.0 RPM (not deb) as of yet, but has been heavily tested on clusters of varying sizes (1 node - 255 node clusters)

          Show
          grimesmi Mike Grimes added a comment - Only includes update to oozie 4.2.0 RPM (not deb) as of yet, but has been heavily tested on clusters of varying sizes (1 node - 255 node clusters)
          Hide
          grimesmi Mike Grimes added a comment -

          Sorry, previous patch did not apply cleanly.

          Show
          grimesmi Mike Grimes added a comment - Sorry, previous patch did not apply cleanly.
          Hide
          grimesmi Mike Grimes added a comment -

          Sorry for noise, I've double checked that this patch applies cleanly this time

          Show
          grimesmi Mike Grimes added a comment - Sorry for noise, I've double checked that this patch applies cleanly this time
          Hide
          evans_ye Evans Ye added a comment -

          Comparing to the YoungWoo Kim's patch, zookeeper has been took off. I guess that's because HA is an advanced feature that doesn't have to be enabled(required) by default? The others LGTM.
          Are you planning to add deb in this patch? Cause this will break deb.

          Show
          evans_ye Evans Ye added a comment - Comparing to the YoungWoo Kim 's patch, zookeeper has been took off. I guess that's because HA is an advanced feature that doesn't have to be enabled(required) by default? The others LGTM. Are you planning to add deb in this patch? Cause this will break deb.
          Hide
          grimesmi Mike Grimes added a comment - - edited

          Yeah, that's true. I'm not very familiar with deb packages, but I think YoungWoo Kim's patch has the deb requirements in it, so if he doesn't mind I can slap those in there, too. You're right on zookeeper - my thinking was that it would be best to allow the user to choose to install zookeeper on their own, as opposed to forcing a dependency on it.

          Show
          grimesmi Mike Grimes added a comment - - edited Yeah, that's true. I'm not very familiar with deb packages, but I think YoungWoo Kim 's patch has the deb requirements in it, so if he doesn't mind I can slap those in there, too. You're right on zookeeper - my thinking was that it would be best to allow the user to choose to install zookeeper on their own, as opposed to forcing a dependency on it.
          Hide
          warwithin YoungWoo Kim added a comment -

          Mike Grimes, Please double check the changes on oozie.spec. BIGTOP-1915-1.patch does not work for me.
          From my original patch:

          +#BIGTOP_PATCH_FILES
           Requires(pre): /usr/sbin/groupadd, /usr/sbin/useradd
           Requires(post): /sbin/chkconfig
           Requires(preun): /sbin/chkconfig, /sbin/service
          @@ -117,7 +122,7 @@ URL: http://oozie.apache.org/
           Group: Development/Libraries
           License: ASL 2.0
           BuildArch: noarch
          -Requires: bigtop-utils >= 0.7
          +Requires: zookeeper, bigtop-utils >= 0.7
           
           
           %description client
          @@ -133,6 +138,8 @@ Requires: bigtop-utils >= 0.7
           %prep
           %setup -n oozie-%{oozie_base_version}
           
          +#BIGTOP_PATCH_COMMANDS
          

          Oozie 4.2 requires a patch from OOZIE-2305 to build against Hive 1.2, #BIGTOP_PATCH_FILES and #BIGTOP_PATCH_COMMANDS are special keywords on Bigtop to apply patches.

          Show
          warwithin YoungWoo Kim added a comment - Mike Grimes , Please double check the changes on oozie.spec. BIGTOP-1915 -1.patch does not work for me. From my original patch: +#BIGTOP_PATCH_FILES Requires(pre): /usr/sbin/groupadd, /usr/sbin/useradd Requires(post): /sbin/chkconfig Requires(preun): /sbin/chkconfig, /sbin/service @@ -117,7 +122,7 @@ URL: http: //oozie.apache.org/ Group: Development/Libraries License: ASL 2.0 BuildArch: noarch -Requires: bigtop-utils >= 0.7 +Requires: zookeeper, bigtop-utils >= 0.7 %description client @@ -133,6 +138,8 @@ Requires: bigtop-utils >= 0.7 %prep %setup -n oozie-%{oozie_base_version} +#BIGTOP_PATCH_COMMANDS Oozie 4.2 requires a patch from OOZIE-2305 to build against Hive 1.2, #BIGTOP_PATCH_FILES and #BIGTOP_PATCH_COMMANDS are special keywords on Bigtop to apply patches.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user youngwookim closed the pull request at:

          https://github.com/apache/bigtop/pull/28

          Show
          githubbot ASF GitHub Bot added a comment - Github user youngwookim closed the pull request at: https://github.com/apache/bigtop/pull/28
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user youngwookim opened a pull request:

          https://github.com/apache/bigtop/pull/58

          BIGTOP-1915. Upgrade Oozie to 4.2.0

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/youngwookim/bigtop BIGTOP-1915

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/58.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #58


          commit 4b344cb1a9bf7c57b10ff1284e9d7562ba1dfc00
          Author: YoungWoo Kim <ywkim@apache.org>
          Date: 2015-07-01T07:22:53Z

          BIGTOP-1915. Upgrade Oozie to 4.2.0


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user youngwookim opened a pull request: https://github.com/apache/bigtop/pull/58 BIGTOP-1915 . Upgrade Oozie to 4.2.0 You can merge this pull request into a Git repository by running: $ git pull https://github.com/youngwookim/bigtop BIGTOP-1915 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/58.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #58 commit 4b344cb1a9bf7c57b10ff1284e9d7562ba1dfc00 Author: YoungWoo Kim <ywkim@apache.org> Date: 2015-07-01T07:22:53Z BIGTOP-1915 . Upgrade Oozie to 4.2.0
          Hide
          warwithin YoungWoo Kim added a comment -

          Mike Grimes, I've just rebased the previous patch and sent a pull request. I believe it's a good starting point to work your own.

          Show
          warwithin YoungWoo Kim added a comment - Mike Grimes , I've just rebased the previous patch and sent a pull request. I believe it's a good starting point to work your own.
          Hide
          grimesmi Mike Grimes added a comment -

          YoungWoo Kim Thanks, so it turns out my modified patch is barely different from yours, and only really includes the change in bigtop.bom, and I took out the requirement for zookeeper. You did all the hard work here so feel free to keep your name on the commit. I was able to test this and get it built and running via the rpm build & centos vm, and was also able to get the .deb to build, but was unable to get ubuntu running in the vm, so was unable to test that it installs correctly.

          Show
          grimesmi Mike Grimes added a comment - YoungWoo Kim Thanks, so it turns out my modified patch is barely different from yours, and only really includes the change in bigtop.bom, and I took out the requirement for zookeeper. You did all the hard work here so feel free to keep your name on the commit. I was able to test this and get it built and running via the rpm build & centos vm, and was also able to get the .deb to build, but was unable to get ubuntu running in the vm, so was unable to test that it installs correctly.
          Hide
          grimesmi Mike Grimes added a comment -

          YoungWoo Kim or Evans Ye able to look at this again to see if we can't get it into the 1.1 release?

          Show
          grimesmi Mike Grimes added a comment - YoungWoo Kim or Evans Ye able to look at this again to see if we can't get it into the 1.1 release?
          Hide
          evans_ye Evans Ye added a comment - - edited

          Hi Mike Grimes
          Yeah, this should be in the 1.1 release.
          The rpms are good during my test, however debian builds are failing.
          The error messages:

          mv: target './debian/oozie/usr/lib/oozie/lib/' is not a directory
          debian/rules:39: recipe for target 'override_dh_auto_install' failed
          make[1]: *** [override_dh_auto_install] Error 1
          make[1]: Leaving directory '/ws/output/oozie/oozie-4.2.0'
          debian/rules:27: recipe for target 'binary' failed
          make: *** [binary] Error 2
          

          You can see detail build log at here.

          Show
          evans_ye Evans Ye added a comment - - edited Hi Mike Grimes Yeah, this should be in the 1.1 release. The rpms are good during my test, however debian builds are failing. The error messages: mv: target './debian/oozie/usr/lib/oozie/lib/' is not a directory debian/rules:39: recipe for target 'override_dh_auto_install' failed make[1]: *** [override_dh_auto_install] Error 1 make[1]: Leaving directory '/ws/output/oozie/oozie-4.2.0' debian/rules:27: recipe for target 'binary' failed make: *** [binary] Error 2 You can see detail build log at here .
          Hide
          grimesmi Mike Grimes added a comment -

          Fixed so DEB package builds, and re-verified that the RPM package still builds after having made the change. Sorry for the back and forth

          Show
          grimesmi Mike Grimes added a comment - Fixed so DEB package builds, and re-verified that the RPM package still builds after having made the change. Sorry for the back and forth
          Hide
          evans_ye Evans Ye added a comment -

          LGTM, +1. Will commit.

          Show
          evans_ye Evans Ye added a comment - LGTM, +1. Will commit.
          Hide
          evans_ye Evans Ye added a comment -

          Committed.
          Thank you both YoungWoo Kim and Mike Grimes.

          Show
          evans_ye Evans Ye added a comment - Committed. Thank you both YoungWoo Kim and Mike Grimes .
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user youngwookim closed the pull request at:

          https://github.com/apache/bigtop/pull/58

          Show
          githubbot ASF GitHub Bot added a comment - Github user youngwookim closed the pull request at: https://github.com/apache/bigtop/pull/58

            People

            • Assignee:
              itsgrimetime Mike Grimes
              Reporter:
              warwithin YoungWoo Kim
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development