Bigtop
  1. Bigtop
  2. BIGTOP-492

make our launcher scripts recognize cascading defaults

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.4.0
    • Fix Version/s: None
    • Component/s: general
    • Labels:
      None

      Description

      Almost all Unix (POSIX?) command line utilities have a nice feature of recognizing cascading defaults. The system-level one is typically named /etc/<util>rc and the user-level one is typically picked from ~/.<util>rc

      I propose that we introduce the same for all of the /usr/bin launcher scripts that we support in Bigtop. The proposed format of the rc files is that of POSIX-compatible sh(1) code snippet with an overall effect of exporting those variables that affect execution.

      Thoughts?

        Issue Links

          Activity

          Hide
          Roman Shaposhnik added a comment - - edited

          Guys, this whole thing is really quite simple. Basically, here's what I propose to add to each of our launcher scripts before it execs the downstream script:

          [ -e /etc/bigtop/<component name>rc ] && . /etc/bigtop/<component name>rc
          [ -e ~/.bigtop/<component name>rc ] && . ~/.bigtop/<component name>rc
          

          Nothing more, nothing less. Very, very simple stuff.

          Show
          Roman Shaposhnik added a comment - - edited Guys, this whole thing is really quite simple. Basically, here's what I propose to add to each of our launcher scripts before it execs the downstream script: [ -e /etc/bigtop/<component name>rc ] && . /etc/bigtop/<component name>rc [ -e ~/.bigtop/<component name>rc ] && . ~/.bigtop/<component name>rc Nothing more, nothing less. Very, very simple stuff.
          Hide
          Bruno Mahé added a comment -

          But in any case users will have to know which variable to export/override, no?
          What layer of abstraction are you thinking about? How much complexity will be added?

          Show
          Bruno Mahé added a comment - But in any case users will have to know which variable to export/override, no? What layer of abstraction are you thinking about? How much complexity will be added?
          Hide
          Roman Shaposhnik added a comment -

          @Philip,

          the goal here is consistency, I guess. I'm sure in every single Hadoop ecosystem component there's a way to achieve this goal of cascading defaults. They are all different though and difficult to remember. A proposed scheme has a nice advantage of regularizing this to a point where you can be sure of what exact shell environment all of the launcher scripts will be executed in.

          As I mentioned, the downside is divergence from upstream.

          Show
          Roman Shaposhnik added a comment - @Philip, the goal here is consistency, I guess. I'm sure in every single Hadoop ecosystem component there's a way to achieve this goal of cascading defaults. They are all different though and difficult to remember. A proposed scheme has a nice advantage of regularizing this to a point where you can be sure of what exact shell environment all of the launcher scripts will be executed in. As I mentioned, the downside is divergence from upstream.
          Hide
          Philip Zeyliger added a comment -

          Roman,

          Seems like HADOOP_CONF_DIR already lets you override stuff; is that not enough?

          Show
          Philip Zeyliger added a comment - Roman, Seems like HADOOP_CONF_DIR already lets you override stuff; is that not enough?
          Hide
          Bruno Mahé added a comment - - edited
          • Could you give more details regarding how you see this being done?
          • How are these scripts created? populated? read?
          • How do you plan on educating users about this Apache Bigtop (incubating) specific feature?
          • How more complicated will become debugging configurations issues?
          • What are the benefits from a user point of view? Just avoiding exporting <COMPONENT>_CONF_DIR>, passing parameter -conf, overriding properties -D<something>, editing the files? Why use that when I can just points to another conf dir?
          • How much complexity/layer of indirection will this feature add? Is the added value worthwhile?

          Overall I am neutral provided this does not add too much confusion/complexity/indirection.
          Although I believe there are a lot more important issues with a lot more value added. But Apache Bigtop (incubating) being a volunteer based project, anyone can work on anything they like

          Note: I prefer /etc/bigtop/<util>rc

          Show
          Bruno Mahé added a comment - - edited Could you give more details regarding how you see this being done? How are these scripts created? populated? read? How do you plan on educating users about this Apache Bigtop (incubating) specific feature? How more complicated will become debugging configurations issues? What are the benefits from a user point of view? Just avoiding exporting <COMPONENT>_CONF_DIR>, passing parameter -conf, overriding properties -D<something>, editing the files? Why use that when I can just points to another conf dir? How much complexity/layer of indirection will this feature add? Is the added value worthwhile? Overall I am neutral provided this does not add too much confusion/complexity/indirection. Although I believe there are a lot more important issues with a lot more value added. But Apache Bigtop (incubating) being a volunteer based project, anyone can work on anything they like Note: I prefer /etc/bigtop/<util>rc
          Hide
          Roman Shaposhnik added a comment -

          This is basically aimed at generalizing <foo>-env.sh. It can be used for having a single client for multiple clusters or anything else where you might want to have a system-level defaults and then
          user-level ones. It is NOT explicitly for *xml. It is a replacement/augmentation for *env.sh

          The trouble with the current Hadoop ecosystem is that it is very developer-centric in its use of <foo>-env.sh:

          1. this script must come from the HOME of the component and not system-level location
          2. this script doesn't allow for a second level of defaults

          Essentially, if you want to override things in, lets say, hadoop-env.sh you can only do it at a system-level leaving the user to use .bashrc for user-level overrides. That's the core issue this JIRA is trying to address.

          Show
          Roman Shaposhnik added a comment - This is basically aimed at generalizing <foo>-env.sh. It can be used for having a single client for multiple clusters or anything else where you might want to have a system-level defaults and then user-level ones. It is NOT explicitly for *xml. It is a replacement/augmentation for *env.sh The trouble with the current Hadoop ecosystem is that it is very developer-centric in its use of <foo>-env.sh: this script must come from the HOME of the component and not system-level location this script doesn't allow for a second level of defaults Essentially, if you want to override things in, lets say, hadoop-env.sh you can only do it at a system-level leaving the user to use .bashrc for user-level overrides. That's the core issue this JIRA is trying to address.
          Hide
          Philip Zeyliger added a comment -

          Is this to more easily support multiple Hadoop clusters on the same machine? Is it for the *xml files, the *env files, both?

          I'm not clear what this issue is trying to fix. .my.cnf, something vaguely similar, lets you specify MySQL client configs, but Hadoop's configs are very mixed client/server.

          Show
          Philip Zeyliger added a comment - Is this to more easily support multiple Hadoop clusters on the same machine? Is it for the *xml files, the *env files, both? I'm not clear what this issue is trying to fix. .my.cnf , something vaguely similar, lets you specify MySQL client configs, but Hadoop's configs are very mixed client/server.
          Hide
          Roman Shaposhnik added a comment -

          I'm not quite sure whether /etc/<util>rc or /etc/bigtop/<util>rc will be a better name for the system-level rc file for each component.

          Show
          Roman Shaposhnik added a comment - I'm not quite sure whether /etc/<util>rc or /etc/bigtop/<util>rc will be a better name for the system-level rc file for each component.

            People

            • Assignee:
              Roman Shaposhnik
              Reporter:
              Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development