Details

    • Type: Task Task
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.0
    • Component/s: None
    • Labels:
      None

      Description

      Projects like Oozie and Sqoop present some configuration challenges compared to other components because they use Tomcat. Sometimes small tweaks to the configuration or classpath have to be done in a very component-specific way as opposed to tweaking files in /etc/<comp>/conf or /etc/default. In one case, we even have redundant Tomcat deployments for common configurations (Oozie's SSL vs. non-SSL).

      If the environment for Tomcat was generated more dynamically, we could avoid this redundancy and could allow common features to be configured in more "standard" ways.

        Issue Links

          Activity

          Hide
          Arvind Prabhakar added a comment -

          Thanks @Sean for filing this issue. I believe having a consistent mechanism for controlling and modifying the system classpath across projects like Sqoop2 and Oozie will greatly facilitate the adoption and ease of management of such systems.

          To that effect, I propose that these modifications be done via catalina.properties exclusively and where necessary things be parameterized via environment variables. Using this configuration file for Tomcat makes sense since it has retained compatibility across major releases of Tomcat whereas the container classpath hierarchy/structures have significantly evolved.

          Show
          Arvind Prabhakar added a comment - Thanks @Sean for filing this issue. I believe having a consistent mechanism for controlling and modifying the system classpath across projects like Sqoop2 and Oozie will greatly facilitate the adoption and ease of management of such systems. To that effect, I propose that these modifications be done via catalina.properties exclusively and where necessary things be parameterized via environment variables. Using this configuration file for Tomcat makes sense since it has retained compatibility across major releases of Tomcat whereas the container classpath hierarchy/structures have significantly evolved.
          Hide
          Sean Mackrory added a comment -

          Thanks for the input Arvind Prabhakar - I agree with that idea. This JIRA will also need to pick up where BIGTOP-811 looks to be leaving off. Rather than statically looking inside /var/lib/bigtop for the classpath in catalina.properties, we should include the contents of BIGTOP_CLASSPATH after it is generated.

          Show
          Sean Mackrory added a comment - Thanks for the input Arvind Prabhakar - I agree with that idea. This JIRA will also need to pick up where BIGTOP-811 looks to be leaving off. Rather than statically looking inside /var/lib/bigtop for the classpath in catalina.properties, we should include the contents of BIGTOP_CLASSPATH after it is generated.
          Hide
          Sean Mackrory added a comment -

          I propose that these modifications be done via catalina.properties exclusively

          Perhaps I'm misunderstanding what you mean but that, but (contrary to my comment above I actually don't think this is possible. Not all the things we need to do can be done from catalina.properties alone. Setting the classpath for Sqoop and Oozie will be done in that file, yes, but then the option of configuring SSL for Oozie requires other files to be replaced like web.xml, etc. Other future options might be the same way.

          Here's what I propose. All Tomcat projects ship with a "template" directory like "/usr/lib/<component>/tomcat-deployment.template.", or perhaps "/etc/<component>/conf/tomcat-deployment.template" (both have pros and cons). Options for supported tweaks need to be added to the defaults file "/etc/default/<component>" (for example "AUX_CLASSPATH=..." and "USE_SSL=true"). When the init script starts the service, it copies the deployment template to another location (/usr/lib/<component>/<current location>), deleting what was there if it exists, sources the defaults file and makes tweaks accordingly, and then launches Tomcat that way it does now.

          Now, not being overly familiar with Tomcat or how users are customizing Sqoop and Oozie now, I expect there may be problems with this simple approach, so please do chime in with your thoughts. Of course, if this is agreeable to everyone, I can start on this immediately. Any thoughts from Jarek Jarcec Cecho or Robert Kanter?

          Show
          Sean Mackrory added a comment - I propose that these modifications be done via catalina.properties exclusively Perhaps I'm misunderstanding what you mean but that, but (contrary to my comment above I actually don't think this is possible. Not all the things we need to do can be done from catalina.properties alone. Setting the classpath for Sqoop and Oozie will be done in that file, yes, but then the option of configuring SSL for Oozie requires other files to be replaced like web.xml, etc. Other future options might be the same way. Here's what I propose. All Tomcat projects ship with a "template" directory like "/usr/lib/<component>/tomcat-deployment.template.", or perhaps "/etc/<component>/conf/tomcat-deployment.template" (both have pros and cons). Options for supported tweaks need to be added to the defaults file "/etc/default/<component>" (for example "AUX_CLASSPATH=..." and "USE_SSL=true"). When the init script starts the service, it copies the deployment template to another location (/usr/lib/<component>/<current location>), deleting what was there if it exists, sources the defaults file and makes tweaks accordingly, and then launches Tomcat that way it does now. Now, not being overly familiar with Tomcat or how users are customizing Sqoop and Oozie now, I expect there may be problems with this simple approach, so please do chime in with your thoughts. Of course, if this is agreeable to everyone, I can start on this immediately. Any thoughts from Jarek Jarcec Cecho or Robert Kanter ?
          Hide
          Robert Kanter added a comment -

          Yes, I also don't think everything can be done in catalina.properties; in fact, looking at Oozie's catalina.properties, I don't think we've changed anything from the defaults in there.

          Sean Mackrory's approach sounds reasonable to me: start with a default/template tomcat and then apply tweaks on top of it depending on the configuration needed. For example, the only difference between the SSL Oozie and regular Oozie is two XML files with only minor changes in them; instead of duplicating the entire tomcat, it would be better to replace the two files at startup (this is also similar to how Oozie switches between SSL and regular).

          Show
          Robert Kanter added a comment - Yes, I also don't think everything can be done in catalina.properties; in fact, looking at Oozie's catalina.properties, I don't think we've changed anything from the defaults in there. Sean Mackrory 's approach sounds reasonable to me: start with a default/template tomcat and then apply tweaks on top of it depending on the configuration needed. For example, the only difference between the SSL Oozie and regular Oozie is two XML files with only minor changes in them; instead of duplicating the entire tomcat, it would be better to replace the two files at startup (this is also similar to how Oozie switches between SSL and regular).
          Hide
          Jarek Jarcec Cecho added a comment -

          Hi Sean Mackrory, thank you very much for bringing this JIRA to my attention. I currently do not see any issues with that, so please go ahead!

          Show
          Jarek Jarcec Cecho added a comment - Hi Sean Mackrory , thank you very much for bringing this JIRA to my attention. I currently do not see any issues with that, so please go ahead!
          Hide
          Roman Shaposhnik added a comment -

          Sean Mackrory in general your proposal looks fine, there's one significant point though: once we start generating configs on the fly they have to be migrated to under /var/lib. /usr/lib is considered immutable for all practical purposes.

          Show
          Roman Shaposhnik added a comment - Sean Mackrory in general your proposal looks fine, there's one significant point though: once we start generating configs on the fly they have to be migrated to under /var/lib. /usr/lib is considered immutable for all practical purposes.
          Hide
          Bruno Mahé added a comment -

          As Roman pointed out, /usr shall be considered as immutable.
          Also people may want to tweak these templates. /usr/lib would be a bad place for people to edit files. Therefore, I would rather have these template in /etc.

          Also this is not the first time I see a ticket about having templates, be it in builds or deployments. Would it be helpful if we start thinking about using a common templating engine for most of our use cases?

          Show
          Bruno Mahé added a comment - As Roman pointed out, /usr shall be considered as immutable. Also people may want to tweak these templates. /usr/lib would be a bad place for people to edit files. Therefore, I would rather have these template in /etc. Also this is not the first time I see a ticket about having templates, be it in builds or deployments. Would it be helpful if we start thinking about using a common templating engine for most of our use cases?
          Hide
          Sean Mackrory added a comment -

          Thanks for the comments, everyone. I agree with the points about using /etc for the templates and /var/lib for the dynamic deployments - I will use those locations in my implementation.

          Also this is not the first time I see a ticket about having templates, be it in builds or deployments. Would it be helpful if we start thinking about using a common templating engine for most of our use cases?

          I would support that.

          Show
          Sean Mackrory added a comment - Thanks for the comments, everyone. I agree with the points about using /etc for the templates and /var/lib for the dynamic deployments - I will use those locations in my implementation. Also this is not the first time I see a ticket about having templates, be it in builds or deployments. Would it be helpful if we start thinking about using a common templating engine for most of our use cases? I would support that.
          Hide
          Roman Shaposhnik added a comment -

          Sean Mackrory this is an extremely good point. Would you mind creating (and driving ) a separate JIRA to make it happen in Bigtop 0.7.0 ?

          Show
          Roman Shaposhnik added a comment - Sean Mackrory this is an extremely good point. Would you mind creating (and driving ) a separate JIRA to make it happen in Bigtop 0.7.0 ?
          Hide
          Sean Mackrory added a comment -

          Attaching a proof-of-concept patch. I've installed both RPMs and DEBs, and the configuration files are in /etc (the webapps directory which contains the WAR and some other configuration files is still in /usr/lib), and are copied to /var/lib at run-time. I still need to test a few specific features this is meant to help support, but just attaching now in case there's any early feedback.

          Show
          Sean Mackrory added a comment - Attaching a proof-of-concept patch. I've installed both RPMs and DEBs, and the configuration files are in /etc (the webapps directory which contains the WAR and some other configuration files is still in /usr/lib), and are copied to /var/lib at run-time. I still need to test a few specific features this is meant to help support, but just attaching now in case there's any early feedback.
          Hide
          Sean Mackrory added a comment -

          I've tested that this makes it much nicer to enable HTTPS/SSL in Oozie, and that this will also make for a more dynamic solution to BIGTOP-811 as well. Submitting patch for review!

          Show
          Sean Mackrory added a comment - I've tested that this makes it much nicer to enable HTTPS/SSL in Oozie, and that this will also make for a more dynamic solution to BIGTOP-811 as well. Submitting patch for review!
          Hide
          Sean Mackrory added a comment -

          (Also - I intend to do this for HTTPFS and Solr as well - but I'll wait for a bit of feedback on what I have, because it should be essentially the same).

          Show
          Sean Mackrory added a comment - (Also - I intend to do this for HTTPFS and Solr as well - but I'll wait for a bit of feedback on what I have, because it should be essentially the same).
          Hide
          Sean Mackrory added a comment -

          Added support for Solr and HTTPFS. Tested service start-up and status-checks on both Ubuntu and Red Hat.

          Show
          Sean Mackrory added a comment - Added support for Solr and HTTPFS. Tested service start-up and status-checks on both Ubuntu and Red Hat.
          Hide
          Bruno Mahé added a comment -

          I am still confused regarding what problem this ticket is trying to solve.
          So some questions:

          • What configuration challenges did you hit?
          • Why can't these configurations be generated at build time?
          • Configurations don't change most of the time. So why regenerating them for every start?
          • As a user, how do I change tweak configurations without messing up packages (ie. editing files in /usr)?
          • Why do we have specific use cases such as https coded in the init script in bash?

          I don't mind generating files from templates, but I would rather have this step at build time of packages.

          Show
          Bruno Mahé added a comment - I am still confused regarding what problem this ticket is trying to solve. So some questions: What configuration challenges did you hit? Why can't these configurations be generated at build time? Configurations don't change most of the time. So why regenerating them for every start? As a user, how do I change tweak configurations without messing up packages (ie. editing files in /usr)? Why do we have specific use cases such as https coded in the init script in bash? I don't mind generating files from templates, but I would rather have this step at build time of packages.
          Hide
          Sean Mackrory added a comment -

          There's a couple of specific problems this was inspired by, but the more general motivation is your 4th point: currently all the Tomcat configuration is under /usr. This patch moves most of it to /etc so if you need to tweak it in an unanticipated way, your changes are honored by the package manager. Now for some specific use-cases:

          BIGTOP-811. We can add /var/lib/bigtop/* to the classpath at build-time, but as the original intent was to detect the MySQL Java connector if it was installed by packages (in /usr/share/java) and the idea of detecting it and creating symlinks in /var/lib/bigtop was -1'd, this is the only other way I see to dynamically add it to the classpath. Because of Tomcat, Sqoop 1.99.x and Oozie won't be able to pick additional classpath entries from the environment.

          BIGTOP-881. In order to enable HTTPS/SSL, users would have had to make several changes to the configuration under /usr. At the time the solution was to deploy two independent pre-built configurations and have the user choose the one they wanted Tomcat to use in the defaults file. With this patch, the configuration can be edited, there's only a single of configs, and that feature is very easy to turn on. I feel like SSL is a common enough feature that supporting the relevant config changes out of the box would be nice, but I'm not 100% married to the idea. At the very least we ought to move the configs to /etc and stop shipping 2 pre-built configs, though.

          There are already more examples of this kind of thing in a certain distribution downstream of Bigtop, which leads me to believe these kinds of things will likely continue to come up. In very general terms, I think having Tomcat applications manage their configuration this way will allow Bigtop to continue making the experience easier without having to get one's hands dirty in the embedded Tomcat stuff.

          Show
          Sean Mackrory added a comment - There's a couple of specific problems this was inspired by, but the more general motivation is your 4th point: currently all the Tomcat configuration is under /usr. This patch moves most of it to /etc so if you need to tweak it in an unanticipated way, your changes are honored by the package manager. Now for some specific use-cases: BIGTOP-811 . We can add /var/lib/bigtop/* to the classpath at build-time, but as the original intent was to detect the MySQL Java connector if it was installed by packages (in /usr/share/java) and the idea of detecting it and creating symlinks in /var/lib/bigtop was -1'd, this is the only other way I see to dynamically add it to the classpath. Because of Tomcat, Sqoop 1.99.x and Oozie won't be able to pick additional classpath entries from the environment. BIGTOP-881 . In order to enable HTTPS/SSL, users would have had to make several changes to the configuration under /usr. At the time the solution was to deploy two independent pre-built configurations and have the user choose the one they wanted Tomcat to use in the defaults file. With this patch, the configuration can be edited, there's only a single of configs, and that feature is very easy to turn on. I feel like SSL is a common enough feature that supporting the relevant config changes out of the box would be nice, but I'm not 100% married to the idea. At the very least we ought to move the configs to /etc and stop shipping 2 pre-built configs, though. There are already more examples of this kind of thing in a certain distribution downstream of Bigtop, which leads me to believe these kinds of things will likely continue to come up. In very general terms, I think having Tomcat applications manage their configuration this way will allow Bigtop to continue making the experience easier without having to get one's hands dirty in the embedded Tomcat stuff.
          Hide
          Bruno Mahé added a comment -

          There are already more examples of this kind of thing in a certain distribution downstream of Bigtop, which leads me to believe these kinds of things will likely continue to come up.

          Your patch makes sense, but this becoming a pattern is exactly why I am trying to be more careful.
          I will try to build a package with your patch and look more into it.

          In the mean time:

          • Why not having a static symlink to /etc and have people use alternatives to switch between ssh/no-ssl configs?
          • +  if [ -e ${DEPLOYMENT_TARGET} ] ; then
            +      rm -r ${DEPLOYMENT_TARGET}
            +  fi
            

            Shouldn't the rm command have also a -f parameter?

          Show
          Bruno Mahé added a comment - There are already more examples of this kind of thing in a certain distribution downstream of Bigtop, which leads me to believe these kinds of things will likely continue to come up. Your patch makes sense, but this becoming a pattern is exactly why I am trying to be more careful. I will try to build a package with your patch and look more into it. In the mean time: Why not having a static symlink to /etc and have people use alternatives to switch between ssh/no-ssl configs? + if [ -e ${DEPLOYMENT_TARGET} ] ; then + rm -r ${DEPLOYMENT_TARGET} + fi Shouldn't the rm command have also a -f parameter?
          Hide
          Sean Mackrory added a comment -

          Why not having a static symlink to /etc and have people use alternatives to switch between ssh/no-ssl configs?

          Mainly just because I'd like to avoid shipping multiple sets of configs. I think it's a necessity with the SSL situation because users shouldn't edit stuff in /usr, but if we really don't like the way I've done the SSL support, I think I'd rather have users make the necessary changes in /etc themselves than use alternatives - but yeah that would certainly be technically doable.

          Shouldn't the rm command have also a -f parameter?

          Yeah. Hasn't caused a problem for me yet, but that's probably good to have. Will add to the patch...

          Show
          Sean Mackrory added a comment - Why not having a static symlink to /etc and have people use alternatives to switch between ssh/no-ssl configs? Mainly just because I'd like to avoid shipping multiple sets of configs. I think it's a necessity with the SSL situation because users shouldn't edit stuff in /usr, but if we really don't like the way I've done the SSL support, I think I'd rather have users make the necessary changes in /etc themselves than use alternatives - but yeah that would certainly be technically doable. Shouldn't the rm command have also a -f parameter? Yeah. Hasn't caused a problem for me yet, but that's probably good to have. Will add to the patch...
          Hide
          Sean Mackrory added a comment -

          Bruno Mahé: Also, just to be clear about BIGTOP-811, depending on whether that or this is committed first - I would eventually want to add the ability for Sqoop and Oozie to dynamic add the contents of the BIGTOP_CLASSPATH environment variable to the classpath in catalina.properties. I haven't added it to the patch because BIGTOP-811 is also still being reviewed, but that's what I'm going for, if you have any thoughts on that.

          Show
          Sean Mackrory added a comment - Bruno Mahé : Also, just to be clear about BIGTOP-811 , depending on whether that or this is committed first - I would eventually want to add the ability for Sqoop and Oozie to dynamic add the contents of the BIGTOP_CLASSPATH environment variable to the classpath in catalina.properties. I haven't added it to the patch because BIGTOP-811 is also still being reviewed, but that's what I'm going for, if you have any thoughts on that.
          Hide
          Sean Mackrory added a comment -

          Someone asked about Reviewboard so it would be easier to comment on specific lines: https://reviews.apache.org/r/12526/

          Show
          Sean Mackrory added a comment - Someone asked about Reviewboard so it would be easier to comment on specific lines: https://reviews.apache.org/r/12526/
          Hide
          Mark Grover added a comment -

          Sean, thanks for working on this! I left some comments on reviewboard.

          Show
          Mark Grover added a comment - Sean, thanks for working on this! I left some comments on reviewboard.
          Hide
          Sean Mackrory added a comment -

          Bruno Mahé - the more I think about it, the more I think your suggestion to use alternatives to switch between secure / insecure configurations is the correct one. The tricky thing is that it also requires a configuration change under /webapps/, which also contains binaries and shouldn't be completely inside /etc, so we'll need to split the configuration out of that as well. I'll see how neatly I can organize all that.

          Show
          Sean Mackrory added a comment - Bruno Mahé - the more I think about it, the more I think your suggestion to use alternatives to switch between secure / insecure configurations is the correct one. The tricky thing is that it also requires a configuration change under /webapps/, which also contains binaries and shouldn't be completely inside /etc, so we'll need to split the configuration out of that as well. I'll see how neatly I can organize all that.
          Hide
          Bruno Mahé added a comment -

          Another option could also be about providing a utility/script to switch between these configs.
          This tool would abstract the user from alternatives/symlinks/etc.

          Show
          Bruno Mahé added a comment - Another option could also be about providing a utility/script to switch between these configs. This tool would abstract the user from alternatives/symlinks/etc.
          Hide
          Sean Mackrory added a comment -

          So what do you think of this approach? Because I quite like it. As you suggested, it uses an alternatives symlink to switch between the "default" configuration and the "secure" configuration. Also, there were still some configuration files under /usr/lib/oozie/webapps.../WEB-INF, but I've also moved that to be stored under /etc and found at runtime under /var.

          I will still do a bit more testing and cleanup of this, and probably relocate WEB-INF files for other services, but I think what I've done for Oozie addresses all of Bruno's concerns, so I wanted to see if there were any other thoughts on the approach.

          Show
          Sean Mackrory added a comment - So what do you think of this approach? Because I quite like it. As you suggested, it uses an alternatives symlink to switch between the "default" configuration and the "secure" configuration. Also, there were still some configuration files under /usr/lib/oozie/webapps.../WEB-INF, but I've also moved that to be stored under /etc and found at runtime under /var. I will still do a bit more testing and cleanup of this, and probably relocate WEB-INF files for other services, but I think what I've done for Oozie addresses all of Bruno's concerns, so I wanted to see if there were any other thoughts on the approach.
          Hide
          Bruno Mahé added a comment -

          Sounds good to me. I like it too.

          Show
          Bruno Mahé added a comment - Sounds good to me. I like it too.
          Hide
          Sean Mackrory added a comment -

          I actually just realized that the other services have additional JARs and classes underneath WEB-INF, so splitting the configuration into /etc/ and keeping binaries in /lib/ is extremely cumbersome. I think I will leave them as-is, and just have Oozie split it up that way, since it's a rather common step in the configuration that requires this. I'll make my patch consistent across RPM and DEB packages, do a bit more testing and then re-post what I think I would like to commit if no other concerns come up.

          Show
          Sean Mackrory added a comment - I actually just realized that the other services have additional JARs and classes underneath WEB-INF, so splitting the configuration into /etc/ and keeping binaries in /lib/ is extremely cumbersome. I think I will leave them as-is, and just have Oozie split it up that way, since it's a rather common step in the configuration that requires this. I'll make my patch consistent across RPM and DEB packages, do a bit more testing and then re-post what I think I would like to commit if no other concerns come up.
          Hide
          Sean Mackrory added a comment -

          So I've tested that the Oozie and Sqoop servers can start and serve clients on both Debian and Red Hat platforms. I've only made large changes to Oozie since my last round of more extensive testing. I've moved all of the configuration under /etc, and tested that you can toggle between the default and secure configurations with "alternatives --config oozie-tomcat-conf".

          All binaries and the entire webapps directory for other components are still under /usr/lib. I'm happy with how the patch is and believe I've satisified all other feedback so far, so I'm submitting it for proper review...

          Show
          Sean Mackrory added a comment - So I've tested that the Oozie and Sqoop servers can start and serve clients on both Debian and Red Hat platforms. I've only made large changes to Oozie since my last round of more extensive testing. I've moved all of the configuration under /etc, and tested that you can toggle between the default and secure configurations with "alternatives --config oozie-tomcat-conf". All binaries and the entire webapps directory for other components are still under /usr/lib. I'm happy with how the patch is and believe I've satisified all other feedback so far, so I'm submitting it for proper review...
          Hide
          Sean Mackrory added a comment -

          Also, I should add the in the near future I intend to add functionality to the tomcat_deploy functions to add the contents of the BIGTOP_CLASSPATH environment variable to the common.loader line in catalina.properties to make BIGTOP-811 benefit Tomcat components as well. This will be a much cleaner action than what I initially proposed here with the USE_HTTPS variable, but if you believe you will have any objections to that, I'd appreciate you speaking up now since that is one of my intentions in reorganizing the Tomcat configuration.

          Show
          Sean Mackrory added a comment - Also, I should add the in the near future I intend to add functionality to the tomcat_deploy functions to add the contents of the BIGTOP_CLASSPATH environment variable to the common.loader line in catalina.properties to make BIGTOP-811 benefit Tomcat components as well. This will be a much cleaner action than what I initially proposed here with the USE_HTTPS variable, but if you believe you will have any objections to that, I'd appreciate you speaking up now since that is one of my intentions in reorganizing the Tomcat configuration.
          Hide
          Mark Grover added a comment -

          Thanks for uploading the second review at https://reviews.apache.org/r/13943/

          Added a few minor nitpicks. In general, this looks good to me and thanks for doing this! Let's give a day or so in case anyone else has any other comments but life is looking good!

          Show
          Mark Grover added a comment - Thanks for uploading the second review at https://reviews.apache.org/r/13943/ Added a few minor nitpicks. In general, this looks good to me and thanks for doing this! Let's give a day or so in case anyone else has any other comments but life is looking good!
          Hide
          Mark Grover added a comment -

          To be explicit, I am +1 barring discussion of the 2 nitpicks I raised on the review

          Show
          Mark Grover added a comment - To be explicit, I am +1 barring discussion of the 2 nitpicks I raised on the review
          Hide
          Bruno Mahé added a comment -

          +1
          Great job!

          Show
          Bruno Mahé added a comment - +1 Great job!

            People

            • Assignee:
              Sean Mackrory
              Reporter:
              Sean Mackrory
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development