Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1581

Allow multiple Flume agents to be executed as a service using Bigtop init.d script

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: Init scripts
    • Labels:

      Description

      Using the current init.d script only one Flume agent can be managed as a service. In practice there will be more than one Flume agent on a node which need be managed. If the init.d script can be improved to accept a agent name for the service requests (start|stop|restart|...) and manage them it will help many installations.

      1. BIGTOP-1581-0.patch
        6 kB
        Biju Nair
      2. BIGTOP-1581-1.patch
        7 kB
        Biju Nair
      3. BIGTOP-1581-2.patch
        7 kB
        Biju Nair
      4. flume-agent.init
        7 kB
        Biju Nair

        Activity

        Hide
        gsbiju Biju Nair added a comment -

        Updated init.d script to support multiple flume agents to be run as a service.

        Show
        gsbiju Biju Nair added a comment - Updated init.d script to support multiple flume agents to be run as a service.
        Hide
        gsbiju Biju Nair added a comment -

        One approach we are taking is to

        • Allow users to pass an agent name to perform service request (start|stop|restart|...)
          service flume-agent start agent-name
        • flume-agent-name.conf is expected to be stored in $FLUME_CONF_DIR (default /etc/flume/conf directory) for the request to be successful
        • If user doesn't provide an agent-name for a service request (start|stop|restart|...) then service action is taken on all the agents with conf file in FLUME_CONF_DIR.

        Attached is the updated init.d script. If this approach is useful to broader community let me know and I can provide a patch.

        Show
        gsbiju Biju Nair added a comment - One approach we are taking is to Allow users to pass an agent name to perform service request (start|stop|restart|...) service flume-agent start agent-name flume-agent-name.conf is expected to be stored in $FLUME_CONF_DIR (default /etc/flume/conf directory) for the request to be successful If user doesn't provide an agent-name for a service request (start|stop|restart|...) then service action is taken on all the agents with conf file in FLUME_CONF_DIR. Attached is the updated init.d script. If this approach is useful to broader community let me know and I can provide a patch.
        Hide
        rvs Roman Shaposhnik added a comment -

        This looks useful, but before we can commit this, could you please elaborate on what are the typical use cases of running multiple flume agents on the same host? My biggest concern is wasting resources on JVM per agent.

        Show
        rvs Roman Shaposhnik added a comment - This looks useful, but before we can commit this, could you please elaborate on what are the typical use cases of running multiple flume agents on the same host? My biggest concern is wasting resources on JVM per agent.
        Hide
        gsbiju Biju Nair added a comment -

        One of our use case is to copy multiple log files from the same node into corresponding HDFS files and be able to manage them as separate services i.e stop|start the services copying each of the log files. Since the nodes have configuration like 256 GB RAM, resource is not a concern.

        Since the changes made to the init.d script includes the option to start one flume agent by name, users can use that option to bring up a flume agent without wasting the resources if that is the concern.

        But with the current script, users who need to bring up multiple flume-agents need to duplicate the script to manage them as different services. This can also end up polluting the /etc/init.d directory with many init.d scripts.

        Show
        gsbiju Biju Nair added a comment - One of our use case is to copy multiple log files from the same node into corresponding HDFS files and be able to manage them as separate services i.e stop|start the services copying each of the log files. Since the nodes have configuration like 256 GB RAM, resource is not a concern. Since the changes made to the init.d script includes the option to start one flume agent by name, users can use that option to bring up a flume agent without wasting the resources if that is the concern. But with the current script, users who need to bring up multiple flume-agents need to duplicate the script to manage them as different services. This can also end up polluting the /etc/init.d directory with many init.d scripts.
        Hide
        mackrorysd Sean Mackrory added a comment -

        This is reminiscent of BIGTOP-732 - where I essentially did the same thing for HBase RegionServers as running multiple RegionServers per DataNode was sometimes recommended. It may be helpful to browse that thread and see some of the edge cases and complexities that eventually went into that patch.

        Other than the HBase devs who originally suggested that change, I don't know how commonly it's used and it did add a lot of complexity that's going to make things hard if / when we start supporting systemd or another SysV-init replacement - so I would personally be a little cautious about doing this for another service. But I wouldn't -1 it...

        Show
        mackrorysd Sean Mackrory added a comment - This is reminiscent of BIGTOP-732 - where I essentially did the same thing for HBase RegionServers as running multiple RegionServers per DataNode was sometimes recommended. It may be helpful to browse that thread and see some of the edge cases and complexities that eventually went into that patch. Other than the HBase devs who originally suggested that change, I don't know how commonly it's used and it did add a lot of complexity that's going to make things hard if / when we start supporting systemd or another SysV-init replacement - so I would personally be a little cautious about doing this for another service. But I wouldn't -1 it...
        Hide
        rvs Roman Shaposhnik added a comment -

        Thanks for the feedback Sean Mackrory! Biju Nair please provide a patch so we can take a look at how much more complex will the init.d script get.

        Show
        rvs Roman Shaposhnik added a comment - Thanks for the feedback Sean Mackrory ! Biju Nair please provide a patch so we can take a look at how much more complex will the init.d script get.
        Hide
        gsbiju Biju Nair added a comment -

        Patch for review.

        Show
        gsbiju Biju Nair added a comment - Patch for review.
        Hide
        gsbiju Biju Nair added a comment -

        Patch submitted for Roman Shaposhnik to review.

        Show
        gsbiju Biju Nair added a comment - Patch submitted for Roman Shaposhnik to review.
        Hide
        warwithin YoungWoo Kim added a comment - - edited

        Biju Nair Thanks for the patch. Looks good to me but a minor suggestion here. In real world, I added flume monitoring[1] for each agent. It looks like '... -Dflume.monitoring.type=http -Dflume.monitoring.port=34545 ...'. It would be nice if we can enable monitoring for Flume in the init script.

        1. http://flume.apache.org/FlumeUserGuide.html#monitoring

        Show
        warwithin YoungWoo Kim added a comment - - edited Biju Nair Thanks for the patch. Looks good to me but a minor suggestion here. In real world, I added flume monitoring [1] for each agent. It looks like '... -Dflume.monitoring.type=http -Dflume.monitoring.port=34545 ...'. It would be nice if we can enable monitoring for Flume in the init script. 1. http://flume.apache.org/FlumeUserGuide.html#monitoring
        Hide
        gsbiju Biju Nair added a comment - - edited

        YoungWoo Kim, thanks for the feedback. Will it not be better to set these parameters in flume-env.sh which is sourced by flume-ng script either in $JAVA_OPTS or $FLUME_OPTS variables. This will also keep the init.d simple and straight forward while maintaining consistency of single location where parameters for flume-ng need to be set/defined.

        Show
        gsbiju Biju Nair added a comment - - edited YoungWoo Kim , thanks for the feedback. Will it not be better to set these parameters in flume-env.sh which is sourced by flume-ng script either in $JAVA_OPTS or $FLUME_OPTS variables. This will also keep the init.d simple and straight forward while maintaining consistency of single location where parameters for flume-ng need to be set/defined.
        Hide
        jayunit100 jay vyas added a comment - - edited

        if we decide to go with this, can we also add a smoke test for this feature to demonstrate how it is actually exersized.

        also, not clear what the common use case is as sean suggested, please elaborate and thanks for the patch.

        given the trend towards single purpose machines , containers and so on, i bet the reasons to run multiple flume services per machine will be less relevant ,so might not be the best idea.

        Show
        jayunit100 jay vyas added a comment - - edited if we decide to go with this, can we also add a smoke test for this feature to demonstrate how it is actually exersized. also, not clear what the common use case is as sean suggested, please elaborate and thanks for the patch. given the trend towards single purpose machines , containers and so on, i bet the reasons to run multiple flume services per machine will be less relevant ,so might not be the best idea.
        Hide
        rvs Roman Shaposhnik added a comment -

        In general, I'm OK with the patch. Building up on what jay vyas has suggested, I'd like to request that the behaviour is documented as a comment block in the default conf file that we're installing in /etc. Biju Nair any chance you can do this small update so we can commit this?

        Also, it would be nice to add a smoke test, but perhaps it could be done as a separate JIRA.

        Show
        rvs Roman Shaposhnik added a comment - In general, I'm OK with the patch. Building up on what jay vyas has suggested, I'd like to request that the behaviour is documented as a comment block in the default conf file that we're installing in /etc. Biju Nair any chance you can do this small update so we can commit this? Also, it would be nice to add a smoke test, but perhaps it could be done as a separate JIRA.
        Hide
        warwithin YoungWoo Kim added a comment -

        Biju Nair Makes sense. Thanks.

        Show
        warwithin YoungWoo Kim added a comment - Biju Nair Makes sense. Thanks.
        Hide
        gsbiju Biju Nair added a comment - - edited

        Updated patch with details about the behavior in the default Flume agent conf file as suggested by Roman Shaposhnik.

        Show
        gsbiju Biju Nair added a comment - - edited Updated patch with details about the behavior in the default Flume agent conf file as suggested by Roman Shaposhnik .
        Hide
        jayunit100 jay vyas added a comment -

        thanks . can you please detail how we would modify existing smoke tests in order to confirm this feature is working properly before we commit ?
        Also, Biju Nair - would you like to be a maintainer of flume in bigtop ? If so I can add you to MAINTAINERS.txt.

        Show
        jayunit100 jay vyas added a comment - thanks . can you please detail how we would modify existing smoke tests in order to confirm this feature is working properly before we commit ? Also, Biju Nair - would you like to be a maintainer of flume in bigtop ? If so I can add you to MAINTAINERS.txt.
        Hide
        rvs Roman Shaposhnik added a comment -

        jay vyas would you mind handling it as a separate JIRA? And +[a lot] on MAINTAINERS.txt comment!

        Show
        rvs Roman Shaposhnik added a comment - jay vyas would you mind handling it as a separate JIRA? And + [a lot] on MAINTAINERS.txt comment!
        Hide
        gsbiju Biju Nair added a comment - - edited

        jay vyas, from a quick look at the smoke-test, here is a rough outline on modifying the existing smoke test for Flume

        • Change the file name to flume-agent.conf here.
        • Create flume-env.sh in /etc/flume/conf and export FLUME_JAVAOPTS with value
        • Instead of using flume-ng script here user service command to start flume. service flume-agent start. The assumption is that sysvinit is used and flume-agent service is enabled.
        • During teardown perform service flume-agent stop

        If something is overlooked let me know.

        On the second question, I will be happy to help with maintaining the code.

        Show
        gsbiju Biju Nair added a comment - - edited jay vyas , from a quick look at the smoke-test, here is a rough outline on modifying the existing smoke test for Flume Change the file name to flume-agent.conf here . Create flume-env.sh in /etc/flume/conf and export FLUME_JAVAOPTS with value Instead of using flume-ng script here user service command to start flume. service flume-agent start . The assumption is that sysvinit is used and flume-agent service is enabled. During teardown perform service flume-agent stop If something is overlooked let me know. On the second question, I will be happy to help with maintaining the code.
        Hide
        gsbiju Biju Nair added a comment -

        Updated patch using git format-patch which is the standard used to submit patches to BIGTOP project.

        Show
        gsbiju Biju Nair added a comment - Updated patch using git format-patch which is the standard used to submit patches to BIGTOP project .
        Hide
        cos Konstantin Boudnik added a comment -

        Is anyone working on this? Shall it be committed or moved out of the release?

        Show
        cos Konstantin Boudnik added a comment - Is anyone working on this? Shall it be committed or moved out of the release?
        Hide
        cos Konstantin Boudnik added a comment -

        Any Flume experts can take a look? Bruno Mahé, Roman Shaposhnik ?

        Show
        cos Konstantin Boudnik added a comment - Any Flume experts can take a look? Bruno Mahé , Roman Shaposhnik ?
        Hide
        bmahe Bruno Mahé added a comment -

        Looking at it tonight

        Show
        bmahe Bruno Mahé added a comment - Looking at it tonight
        Hide
        cos Konstantin Boudnik added a comment -

        Anything yet? I am planning to do the RC on the weekend one way or another.

        Show
        cos Konstantin Boudnik added a comment - Anything yet? I am planning to do the RC on the weekend one way or another.
        Hide
        bmahe Bruno Mahé added a comment -

        +1

        Show
        bmahe Bruno Mahé added a comment - +1
        Hide
        cos Konstantin Boudnik added a comment -

        As you know we have certain requirements for the commit messages. They have to include ticket number follow by the '.' and the ticket title. If the contributor has submitted a formatter patch with wrong commit message it needs to be fixed, either by the committer or contributor.

        Show
        cos Konstantin Boudnik added a comment - As you know we have certain requirements for the commit messages. They have to include ticket number follow by the '.' and the ticket title. If the contributor has submitted a formatter patch with wrong commit message it needs to be fixed, either by the committer or contributor.
        Hide
        bmahe Bruno Mahé added a comment -

        Sure. What do you suggest in this case?

        Show
        bmahe Bruno Mahé added a comment - Sure. What do you suggest in this case?
        Hide
        cos Konstantin Boudnik added a comment -

        I suggest to leave it as is. It isn't the end of the world

        Show
        cos Konstantin Boudnik added a comment - I suggest to leave it as is. It isn't the end of the world
        Hide
        gsbiju Biju Nair added a comment -

        Konstantin Boudnik, Didn't know the requirement for the commit message. Will try to get a new one.

        Show
        gsbiju Biju Nair added a comment - Konstantin Boudnik , Didn't know the requirement for the commit message. Will try to get a new one.

          People

          • Assignee:
            gsbiju Biju Nair
            Reporter:
            gsbiju Biju Nair
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development