Flume
  1. Flume
  2. FLUME-966

Migrate lifecycle management to Guava service implemetation

    Details

    • Type: Task Task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v1.0.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently flume uses a home grown lifecycle management mechanism. In order to simplify the code, we want to migrate the current implementation to use Guava libraries' service management implementation.

      1. FLUME-966-alpha.patch
        261 kB
        Juhani Connolly
      2. FLUME-966-0.patch
        6 kB
        Brock Noland
      3. FLUME-966.patch
        22 kB
        Juhani Connolly

        Issue Links

          Activity

          Hide
          Brock Noland added a comment -

          OK, thanks!! It's a big change so we'll see

          Show
          Brock Noland added a comment - OK, thanks!! It's a big change so we'll see
          Brock Noland made changes -
          Assignee Juhani Connolly [ juhanic ] Brock Noland [ brocknoland ]
          Hide
          Juhani Connolly added a comment -

          Go for it!

          Show
          Juhani Connolly added a comment - Go for it!
          Hide
          Brock Noland added a comment -

          OK, are you currently working on this one and if not, do you mind if I try it out?

          Show
          Brock Noland added a comment - OK, are you currently working on this one and if not, do you mind if I try it out?
          Hide
          Juhani Connolly added a comment -

          never really got reviewed so assumed there wasn't much interest in this particular way of doing things. It was certainly rough around the edges and was going to need a fair bit of polish

          Show
          Juhani Connolly added a comment - never really got reviewed so assumed there wasn't much interest in this particular way of doing things. It was certainly rough around the edges and was going to need a fair bit of polish
          Brock Noland made changes -
          Link This issue is related to FLUME-1502 [ FLUME-1502 ]
          Hide
          Brock Noland added a comment -

          Juhani,

          Just curious if you have been merging trunk into this branch?

          Brock

          Show
          Brock Noland added a comment - Juhani, Just curious if you have been merging trunk into this branch? Brock
          Hide
          Juhani Connolly added a comment -

          Still a rough version though all tests now pass, and updated for the latest trunk

          Show
          Juhani Connolly added a comment - Still a rough version though all tests now pass, and updated for the latest trunk
          Juhani Connolly made changes -
          Remote Link This issue links to "Patch review (Web Link)" [ 10204 ]
          Juhani Connolly made changes -
          Attachment FLUME-966-alpha.patch [ 12532175 ]
          Hide
          Juhani Connolly added a comment -

          One way to handle the reconfiguration when dealing with Suppliers.

          This affect most of the framework so comes as a huge patch.

          I've ironed out most of the unit tests, but I'm having issues with FileChannels tests(with the locks) that I'm not sure are related.

          This isn't intended for inclusion now, but I would appreciate if people could have a look at it and think this is worth continuing on(since the changes are pretty far-reaching).

          As mentioned elsewhere, this will probably need a separate branch for polish though I would like to come up with a clear target of when/how it would be merged back with trunk(I'd say after the next release would be good?)

          Show
          Juhani Connolly added a comment - One way to handle the reconfiguration when dealing with Suppliers. This affect most of the framework so comes as a huge patch. I've ironed out most of the unit tests, but I'm having issues with FileChannels tests(with the locks) that I'm not sure are related. This isn't intended for inclusion now, but I would appreciate if people could have a look at it and think this is worth continuing on(since the changes are pretty far-reaching). As mentioned elsewhere, this will probably need a separate branch for polish though I would like to come up with a clear target of when/how it would be merged back with trunk(I'd say after the next release would be good?)
          Juhani Connolly made changes -
          Attachment FLUME-966.patch [ 12531631 ]
          Hide
          Juhani Connolly added a comment -

          Rough implementation of lifecycle management framework for discussion.

          Seems like review board is down, I'll post it up there later.

          Some notes. A lot of this is just personal preference and open to change. One can note looking at the new sink runner that using the guava service api simplifies things a fair bit

          • generally simplifying most stuff
          • no central thread pool. Processes are expected to last a long time,
            so recycling of threads is not a target
          • dropping the current state/desired state model to move to the guava
            model of new/starting/running/stopping/terminated.
          • Supervising something will start it, and unsupervising it will stop it
          • terminating a supervisor will terminate associated service
          • a supervisor will create a new service using a supplier every time
            the old one fails. The supplier is responsible for providing a
            freshly configured service with no state.
          • the component interface is just Service + NamedComponent. I would
            expect Sink/Source runners to extend AbstractExecutionThreadService,
            while "passive" components like channels would extend
            AbstractIdleService. I'll add some base classes later along with
            rudimentary tests
          • a reconfiguration would probably involve unsupervising
            everything(sources->sinks->channels), than configuring an item at a
            time.
          Show
          Juhani Connolly added a comment - Rough implementation of lifecycle management framework for discussion. Seems like review board is down, I'll post it up there later. Some notes. A lot of this is just personal preference and open to change. One can note looking at the new sink runner that using the guava service api simplifies things a fair bit generally simplifying most stuff no central thread pool. Processes are expected to last a long time, so recycling of threads is not a target dropping the current state/desired state model to move to the guava model of new/starting/running/stopping/terminated. Supervising something will start it, and unsupervising it will stop it terminating a supervisor will terminate associated service a supervisor will create a new service using a supplier every time the old one fails. The supplier is responsible for providing a freshly configured service with no state. the component interface is just Service + NamedComponent. I would expect Sink/Source runners to extend AbstractExecutionThreadService, while "passive" components like channels would extend AbstractIdleService. I'll add some base classes later along with rudimentary tests a reconfiguration would probably involve unsupervising everything(sources->sinks->channels), than configuring an item at a time.
          Hide
          Juhani Connolly added a comment -

          I'm working on this atm unless Brock is keen on continuing from his earlier patch. Wanted to post current progress before the weekend but not satisfied with what I have right now.

          Show
          Juhani Connolly added a comment - I'm working on this atm unless Brock is keen on continuing from his earlier patch. Wanted to post current progress before the weekend but not satisfied with what I have right now.
          Juhani Connolly made changes -
          Assignee Juhani Connolly [ juhanic ]
          Juhani Connolly made changes -
          Link This issue relates to FLUME-1257 [ FLUME-1257 ]
          Hide
          Arvind Prabhakar added a comment -

          Thanks for the patch Brock. Can you please post it to review board.

          Show
          Arvind Prabhakar added a comment - Thanks for the patch Brock. Can you please post it to review board.
          Brock Noland made changes -
          Field Original Value New Value
          Attachment FLUME-966-0.patch [ 12515281 ]
          Hide
          Brock Noland added a comment -

          Attached patch is not for committing. Just something I put together while looking at this. It implements a Supervised and Unsupervised FlumeService.

          Show
          Brock Noland added a comment - Attached patch is not for committing. Just something I put together while looking at this. It implements a Supervised and Unsupervised FlumeService.
          Show
          Brock Noland added a comment - Good doc http://code.google.com/p/guava-libraries/wiki/ServiceExplained
          Arvind Prabhakar created issue -

            People

            • Assignee:
              Brock Noland
              Reporter:
              Arvind Prabhakar
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development