Bigtop
  1. Bigtop
  2. BIGTOP-474

improve Oozie integration with Hadoop 0.23

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.4.0
    • Fix Version/s: 0.4.0
    • Component/s: general
    • Labels:
      None

      Description

      Here's a bunch of improvements to the way Oozie is packaged in Bigtop that I would like to suggest based on my conversations so far:

      1. renaming /usr/lib/oozie/oozie-core -> /usr/lib/oozie/libserver (that will give us lib, libtools and libserver and is pretty consistent)
      2. add /var/lib/oozie to the catalina.properties classpath so that users can simply drop jar files there (e.g. for jdbc connectors)
      3. add /etc/hadoop/conf to the oozie-site.xml oozie.service.HadoopAccessorService.hadoop.configurations
      4. get rid of /etc/oozie/conf/hadoop-conf (once #3 is done)
      5. make sure we don't ship docs twice (once in /usr/share/docs/oozie and the other time in webapps)

      I'll attach a patch shortly. But in the meantime, please let me know what do you think.

      1. BIGTOP-474.patch.txt
        4 kB
        Roman Shaposhnik

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        1d 19h 9m 1 Roman Shaposhnik 22/Mar/12 18:06
        Resolved Resolved Closed Closed
        456d 5h 49m 1 Roman Shaposhnik 22/Jun/13 00:55
        Roman Shaposhnik made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Gavin made changes -
        Workflow no-reopen-closed, patch-avail [ 12658822 ] patch-available, re-open possible [ 12665673 ]
        Roman Shaposhnik made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Roman Shaposhnik added a comment -

        @Bruno, not really. Currently client has a flat layout – all the jar files are at the same level but they point to all sorts of places.

        Show
        Roman Shaposhnik added a comment - @Bruno, not really. Currently client has a flat layout – all the jar files are at the same level but they point to all sorts of places.
        Hide
        Bruno Mahé added a comment -
        • Don't you need /usr/lib/hadoop/client/lib/*.jar as well in your common loader?

        If not +1

        Show
        Bruno Mahé added a comment - Don't you need /usr/lib/hadoop/client/lib/*.jar as well in your common loader? If not +1
        Hide
        Roman Shaposhnik added a comment -

        Patch attached. Please let me know if sed is too much. The alternative is to have a private copy of the oozie-site.xml which is not that great since oozie is changing rapidly.

        Show
        Roman Shaposhnik added a comment - Patch attached. Please let me know if sed is too much. The alternative is to have a private copy of the oozie-site.xml which is not that great since oozie is changing rapidly.
        Roman Shaposhnik made changes -
        Attachment BIGTOP-474.patch.txt [ 12519258 ]
        Roman Shaposhnik made changes -
        Attachment BIGTOP-474.patch.txt [ 12519257 ]
        Roman Shaposhnik made changes -
        Attachment BIGTOP-474.patch.txt [ 12519257 ]
        Hide
        Alejandro Abdelnur added a comment -

        +1 on the proposal

        Show
        Alejandro Abdelnur added a comment - +1 on the proposal
        Roman Shaposhnik made changes -
        Field Original Value New Value
        Description Here's a bunch of improvements to the way Oozie is packaged in Bigtop that I would like to suggest based on my conversations so far:
          # renaming /usr/lib/oozie/oozie-server -> /usr/lib/oozie/libserver (that will give us lib, libtools and libserver and is pretty consistent)
          # add /var/lib/oozie to the catalina.properties classpath so that users can simply drop jar files there (e.g. for jdbc connectors)
          # add /etc/hadoop/conf to the oozie-site.xml oozie.service.HadoopAccessorService.hadoop.configurations
          # get rid of /etc/oozie/conf/hadoop-conf (once #3 is done)
          # make sure we don't ship docs twice (once in /usr/share/docs/oozie and the other time in webapps)

        I'll attach a patch shortly. But in the meantime, please let me know what do you think.
        Here's a bunch of improvements to the way Oozie is packaged in Bigtop that I would like to suggest based on my conversations so far:
          # renaming /usr/lib/oozie/oozie-core -> /usr/lib/oozie/libserver (that will give us lib, libtools and libserver and is pretty consistent)
          # add /var/lib/oozie to the catalina.properties classpath so that users can simply drop jar files there (e.g. for jdbc connectors)
          # add /etc/hadoop/conf to the oozie-site.xml oozie.service.HadoopAccessorService.hadoop.configurations
          # get rid of /etc/oozie/conf/hadoop-conf (once #3 is done)
          # make sure we don't ship docs twice (once in /usr/share/docs/oozie and the other time in webapps)

        I'll attach a patch shortly. But in the meantime, please let me know what do you think.
        Roman Shaposhnik created issue -

          People

          • Assignee:
            Roman Shaposhnik
            Reporter:
            Roman Shaposhnik
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development