Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Component/s: None
    • Labels:
      None

      Description

      1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. We should change this location to be somewhere that doesn't require root access.

      2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user.

      3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal.

      1. chukwa
        4 kB
        Eric Yang
      2. CHUKWA-446.patch
        96 kB
        Eric Yang
      3. CHUKWA-446-2.patch
        119 kB
        Eric Yang
      4. CHUKWA-446-4.patch
        5 kB
        Eric Yang

        Activity

        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Chukwa-trunk #330 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/330/ )
        Hide
        Eric Yang added a comment -

        Thanks Guillermo, I just committed the change.

        Show
        Eric Yang added a comment - Thanks Guillermo, I just committed the change.
        Hide
        Eric Yang added a comment -

        Clean up some of the improper reference in the shell script.

        Show
        Eric Yang added a comment - Clean up some of the improper reference in the shell script.
        Hide
        Eric Yang added a comment -

        Stop script handles stop command more gracefully, when there is no command to stop.

        Show
        Eric Yang added a comment - Stop script handles stop command more gracefully, when there is no command to stop.
        Hide
        Eric Yang added a comment -

        Correction to startup and stop scripts.

        Show
        Eric Yang added a comment - Correction to startup and stop scripts.
        Hide
        Eric Yang added a comment -

        There is a problem with start-* scripts where chukwa command is already in chukwa-daemons.sh.

        Show
        Eric Yang added a comment - There is a problem with start-* scripts where chukwa command is already in chukwa-daemons.sh.
        Hide
        Eric Yang added a comment -

        I just committed this. Thanks Ari.

        Show
        Eric Yang added a comment - I just committed this. Thanks Ari.
        Hide
        Ari Rabkin added a comment -

        Let's commit it. Passes my smoke test.

        Show
        Ari Rabkin added a comment - Let's commit it. Passes my smoke test.
        Hide
        Eric Yang added a comment -

        Please review, thanks.

        Show
        Eric Yang added a comment - Please review, thanks.
        Hide
        Eric Yang added a comment -

        The new chukwa command looks like this:

        ./bin/chukwa
        
        Usage: chukwa [--config confdir] COMMAND
        where COMMAND is one of:
          agent         run a Chukwa Agent
          archive       run the Archive Manager
          collector     run a Chukwa Collector
          demux         run the Demux Manager
          dp            run the Post Demux data processors
          hicc          run a HICC Webserver
          droll         run a daily rolling job (deprecated)
          hroll         run a hourly rolling job (deprecated)
          version       print the version
        Utilities:
          backfill      run a back fill data loader utility
          dumpArchive   view an archive file
          dumpRecord    view a record file
          tail          start tailing a file
        Most command print help when invoked w/o parameters.
        
        
        Show
        Eric Yang added a comment - The new chukwa command looks like this: ./bin/chukwa Usage: chukwa [--config confdir] COMMAND where COMMAND is one of: agent run a Chukwa Agent archive run the Archive Manager collector run a Chukwa Collector demux run the Demux Manager dp run the Post Demux data processors hicc run a HICC Webserver droll run a daily rolling job (deprecated) hroll run a hourly rolling job (deprecated) version print the version Utilities: backfill run a back fill data loader utility dumpArchive view an archive file dumpRecord view a record file tail start tailing a file Most command print help when invoked w/o parameters.
        Hide
        Eric Yang added a comment -

        Have a updated version.

        Show
        Eric Yang added a comment - Have a updated version.
        Hide
        Eric Yang added a comment -

        First cut of the clean up.

        • Watchdog removed
        • Database table management removed
        • system metrics moved to ExecAdaptor.
        • Rmoved legacy Torque data loader
        • Removed legacy Node Activity data loader.
        • Removed legacy UserHDFSUsage script. This metering solution is exported by JobSummary log file.
        • Single wrapper script for all chukwa commands.
        Show
        Eric Yang added a comment - First cut of the clean up. Watchdog removed Database table management removed system metrics moved to ExecAdaptor. Rmoved legacy Torque data loader Removed legacy Node Activity data loader. Removed legacy UserHDFSUsage script. This metering solution is exported by JobSummary log file. Single wrapper script for all chukwa commands.
        Hide
        Eric Yang added a comment -

        Clean up shell scripts as agreed in this issue.

        Show
        Eric Yang added a comment - Clean up shell scripts as agreed in this issue.
        Hide
        Eric Yang added a comment -

        +1 for no dbAdmin (database data loader should be deprecated.)
        +1 for removing crontab watch dog.

        Show
        Eric Yang added a comment - +1 for no dbAdmin (database data loader should be deprecated.) +1 for removing crontab watch dog.
        Hide
        Bill Graham added a comment -

        So we're talking about having the general purpose chukwa wrapper script, as well as the start-, stop- scripts in bin? If so I'm also good with this refactoring. I'm particularly interested in the start-, stop- scripts.

        If it's at all feasible, I'd like to be able to able to have options on start-data-processors.sh to 1.) not start dbAdmin; and 2.) to not modify the crontab. We don't use dbAdmin and our rpm install manages crontab entries. Again, a nice to have. Perhaps this is beyond the scope of this refactor.

        Show
        Bill Graham added a comment - So we're talking about having the general purpose chukwa wrapper script, as well as the start- , stop- scripts in bin? If so I'm also good with this refactoring. I'm particularly interested in the start- , stop- scripts. If it's at all feasible, I'd like to be able to able to have options on start-data-processors.sh to 1.) not start dbAdmin; and 2.) to not modify the crontab. We don't use dbAdmin and our rpm install manages crontab entries. Again, a nice to have. Perhaps this is beyond the scope of this refactor.
        Hide
        Ari Rabkin added a comment -

        I'm okay with this refactoring.

        Show
        Ari Rabkin added a comment - I'm okay with this refactoring.
        Hide
        Eric Yang added a comment -

        I am keeping all start-* and stop-* commands in the same directory. Hence, the list of files to stay in bin directory are:

        README
        VERSION
        agent.sh
        collector.sh
        chukwa				
        chukwa-config.sh		
        chukwa-daemon.sh		
        chukwa-daemons.sh		
        slaves.sh
        start-agents.sh
        start-all.sh
        start-collectors.sh
        start-data-processors.sh
        stop-agents.sh
        stop-all.sh
        stop-collectors.sh
        stop-data-processors.sh
        
        Show
        Eric Yang added a comment - I am keeping all start-* and stop-* commands in the same directory. Hence, the list of files to stay in bin directory are: README VERSION agent.sh collector.sh chukwa chukwa-config.sh chukwa-daemon.sh chukwa-daemons.sh slaves.sh start-agents.sh start-all.sh start-collectors.sh start-data-processors.sh stop-agents.sh stop-all.sh stop-collectors.sh stop-data-processors.sh
        Hide
        Ari Rabkin added a comment -

        I think there's a typo on "ChukwaArachiveManager"

        Should be Archive, I think, not Arachive.

        Are we ditching start-agents and start-collectors? I use those...but would be happy if they moved to some auxiliary directory to make bin look less scary.

        Show
        Ari Rabkin added a comment - I think there's a typo on "ChukwaArachiveManager" Should be Archive, I think, not Arachive. Are we ditching start-agents and start-collectors? I use those...but would be happy if they moved to some auxiliary directory to make bin look less scary.
        Hide
        Eric Yang added a comment -

        Consolidate Chukwa shell scripts into a single chukwa command:

        Usage: chukwa [--config confdir] COMMAND
        where COMMAND is one of:
          agent         run a Chukwa Agent
          archive       run the Archive Manager
          collector     run a Chukwa Collector
          demux         run the Demux Manager
          dataloader    run the Post Demux Data Loaders
          hicc          run a HICC Webserver
          version       print the version
        Utilities:
          backfill      run a back fill data loader utility
          dumpArchive   view an archive file
          dumpRecord    view a record file
        Most command print help when invoked w/o parameters.
        

        The long list of shell scripts that didn't make to this list will be deleted, unless someone voice their concerns.

        Show
        Eric Yang added a comment - Consolidate Chukwa shell scripts into a single chukwa command: Usage: chukwa [--config confdir] COMMAND where COMMAND is one of: agent run a Chukwa Agent archive run the Archive Manager collector run a Chukwa Collector demux run the Demux Manager dataloader run the Post Demux Data Loaders hicc run a HICC Webserver version print the version Utilities: backfill run a back fill data loader utility dumpArchive view an archive file dumpRecord view a record file Most command print help when invoked w/o parameters. The long list of shell scripts that didn't make to this list will be deleted, unless someone voice their concerns.
        Hide
        Bill Graham added a comment -

        I'm also fine without using the init.d scripts, since after looking more closely at the bin/stat-*.sh scripts they seem like they'll work better for my needs anyway.

        @Eric +1 on renaming jettyCollector.sh to collector.sh for consistency and following the foo.sh, start-foo.sh and stop-foo.sh pattern.

        If we're going to refactor the start-*.sh scripts I'd like to suggest we add a feature to slaves.sh, where if no hostnames are configured, the default behavior would be to start the daemon on localhost without ssh'ing. This would basically be the same as the wrapped script, just with pid files managed by start/stop wrapper.

        Show
        Bill Graham added a comment - I'm also fine without using the init.d scripts, since after looking more closely at the bin/stat-*.sh scripts they seem like they'll work better for my needs anyway. @Eric +1 on renaming jettyCollector.sh to collector.sh for consistency and following the foo.sh, start-foo.sh and stop-foo.sh pattern. If we're going to refactor the start-*.sh scripts I'd like to suggest we add a feature to slaves.sh, where if no hostnames are configured, the default behavior would be to start the daemon on localhost without ssh'ing. This would basically be the same as the wrapped script, just with pid files managed by start/stop wrapper.
        Hide
        Eric Yang added a comment -

        Hadoop has a fat shell script which encapsulated most of the shell script logic. It's possible to mimic the logic:

        Usage: chukwa [--config confdir] COMMAND
        where COMMAND is one of:
        collector run a Chukwa Collector
        agent run a Chukwa Agent
        dp run the data processors
        hicc run the HICC console
        version print the version
        Most commands print help when invoked w/o parameters.

        Show
        Eric Yang added a comment - Hadoop has a fat shell script which encapsulated most of the shell script logic. It's possible to mimic the logic: Usage: chukwa [--config confdir] COMMAND where COMMAND is one of: collector run a Chukwa Collector agent run a Chukwa Agent dp run the data processors hicc run the HICC console version print the version Most commands print help when invoked w/o parameters.
        Hide
        Jerome Boulon added a comment -

        My guess is that 80% of these scripts can be merge into 1 script, then it should just be a matter of passing some parameters but most of the logic should be shared.

        Show
        Jerome Boulon added a comment - My guess is that 80% of these scripts can be merge into 1 script, then it should just be a matter of passing some parameters but most of the logic should be shared.
        Hide
        Eric Yang added a comment -

        start-agent.sh, and stop-agent.sh could be boot trap script for agent.sh and same model applies to jettyCollector.sh. This may be too many wrapper scripts. Could we rename jettyCollector.sh to collector.sh? Hence, for singular use agent.sh and collector.sh, for multiple machines, use start-agents.sh, and start-collectors.sh. I am not sure if this makes sense for everyone. We could vote on this.

        Show
        Eric Yang added a comment - start-agent.sh, and stop-agent.sh could be boot trap script for agent.sh and same model applies to jettyCollector.sh. This may be too many wrapper scripts. Could we rename jettyCollector.sh to collector.sh? Hence, for singular use agent.sh and collector.sh, for multiple machines, use start-agents.sh, and start-collectors.sh. I am not sure if this makes sense for everyone. We could vote on this.
        Hide
        Ari Rabkin added a comment -

        I'm okay just ditching them.

        At Berkeley, we use agent.sh and jettyCollector.sh, managed by runit.

        Show
        Ari Rabkin added a comment - I'm okay just ditching them. At Berkeley, we use agent.sh and jettyCollector.sh, managed by runit.
        Hide
        Eric Yang added a comment -

        The scripts were built to run in a specific environment to make it easy for a black magic grown system. My recommendation is completely remove those scripts and use start-.sh stop-.sh.

        *chukwa-system-metrics, this should be replaced with exec adaptor and configure through initial_adaptors.template.

        • chukwa-agent - rename to start-agent.sh and stop-agent.sh.
        • chukwa-collector - rename to start-collector.sh and stop-collector.sh
        • chukwa-data-processors - use start-data-processors.sh and stop-data-processor.sh
        Show
        Eric Yang added a comment - The scripts were built to run in a specific environment to make it easy for a black magic grown system. My recommendation is completely remove those scripts and use start- .sh stop- .sh. *chukwa-system-metrics, this should be replaced with exec adaptor and configure through initial_adaptors.template. chukwa-agent - rename to start-agent.sh and stop-agent.sh. chukwa-collector - rename to start-collector.sh and stop-collector.sh chukwa-data-processors - use start-data-processors.sh and stop-data-processor.sh
        Hide
        Jerome Boulon added a comment -

        CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER are configurable.
        It's done when you generate the RPM, look at the ANT RPM target and default.properties file.
        To overwrite these values, just create a new file build.properties at the same level.

        Show
        Jerome Boulon added a comment - CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER are configurable. It's done when you generate the RPM, look at the ANT RPM target and default.properties file. To overwrite these values, just create a new file build.properties at the same level.

          People

          • Assignee:
            Eric Yang
            Reporter:
            Bill Graham
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development