Flume
  1. Flume
  2. FLUME-1522

System hook before initialization of all components

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v1.2.0
    • Fix Version/s: None
    • Component/s: Node
    • Labels:
      None

      Description

      Both of Source and Sink are depend on HTTP server. Thus, HTTP server should starts before those components.
      If Flume can support system hook before initialization of all components, we can expand common functions such as monitoring, HTTP server, and so on.
      It just a proposal, and need to discuss.
      This function is useful for me.

        Activity

        Hide
        Mike Percy added a comment -

        Hi Denny, thanks for opening this issue. Can you please clarify regarding the HTTP server? The only HTTP server I am aware of is the JSON monitoring functionality. More details on the use case would also be helpful.

        Show
        Mike Percy added a comment - Hi Denny, thanks for opening this issue. Can you please clarify regarding the HTTP server? The only HTTP server I am aware of is the JSON monitoring functionality. More details on the use case would also be helpful.
        Hide
        Denny Ye added a comment -

        Mike, thanks for your reply. The necessary requirement of HTTP server has raised from my company and team. We need RESTFul Source and Sink around Flume agent. The heartbeat checking and monitoring component based on web UI are also through using HTTP server.
        Most of thus requirements that I could work might be comes from internal. However, the system hook is common design, in my opinion

        Show
        Denny Ye added a comment - Mike, thanks for your reply. The necessary requirement of HTTP server has raised from my company and team. We need RESTFul Source and Sink around Flume agent. The heartbeat checking and monitoring component based on web UI are also through using HTTP server. Most of thus requirements that I could work might be comes from internal. However, the system hook is common design, in my opinion
        Hide
        Mike Percy added a comment -

        Denny, that HTTP REST service sounds like a useful feature for Flume. Have you filed a JIRA about that? If you would like to discuss the design, I would be interested.

        Show
        Mike Percy added a comment - Denny, that HTTP REST service sounds like a useful feature for Flume. Have you filed a JIRA about that? If you would like to discuss the design, I would be interested.
        Hide
        Denny Ye added a comment -

        Mike, what's your suggestion about this proposal?
        I am glad to post JIRA and patch if REST service is useful for others.

        Show
        Denny Ye added a comment - Mike, what's your suggestion about this proposal? I am glad to post JIRA and patch if REST service is useful for others.
        Hide
        Mubarak Seyed added a comment -

        There is a related JIRA for Http Post Source

        Show
        Mubarak Seyed added a comment - There is a related JIRA for Http Post Source
        Hide
        Denny Ye added a comment -

        This JIRA is focusing on system hook function. Would you give me feedback about this proposal?

        Show
        Denny Ye added a comment - This JIRA is focusing on system hook function. Would you give me feedback about this proposal?
        Hide
        Mike Percy added a comment -

        Thanks Mubarak for the pointer to the HTTP POST Source JIRA.

        Denny, I am not sure what the problem is that you're trying to solve. The primary hooks we have today come in the form of the Source, Channel, and Sink interfaces. These components are all pluggable. What do you feel is missing?

        Show
        Mike Percy added a comment - Thanks Mubarak for the pointer to the HTTP POST Source JIRA. Denny, I am not sure what the problem is that you're trying to solve. The primary hooks we have today come in the form of the Source, Channel, and Sink interfaces. These components are all pluggable. What do you feel is missing?
        Hide
        Denny Ye added a comment -

        hi Mike,
        Currently, flume only supports Source, Channel, Sink interfaces and extensions. How to manage the lifecycle of other extensions except listed above?

        Both HttpEventSource and monitoring component are dependent on HTTP server. HTTP server should starts before those components. Thus, I want to use system hook to start the HTTP server and monitoring server before initialization of other components. This is original requirement

        Show
        Denny Ye added a comment - hi Mike, Currently, flume only supports Source, Channel, Sink interfaces and extensions. How to manage the lifecycle of other extensions except listed above? Both HttpEventSource and monitoring component are dependent on HTTP server. HTTP server should starts before those components. Thus, I want to use system hook to start the HTTP server and monitoring server before initialization of other components. This is original requirement
        Hide
        Mike Percy added a comment -

        So what's the difference between an HttpEventSource and an HTTP Server? Are you saying you want to run multiple components that do different things on the same port? If so, why not just run everything on separate ports?

        Show
        Mike Percy added a comment - So what's the difference between an HttpEventSource and an HTTP Server? Are you saying you want to run multiple components that do different things on the same port? If so, why not just run everything on separate ports?
        Hide
        Denny Ye added a comment -

        How to start HTTP monitoring component in Flume? This component dependent on HTTP server.

        Show
        Denny Ye added a comment - How to start HTTP monitoring component in Flume? This component dependent on HTTP server.
        Hide
        Mike Percy added a comment -

        Well we already have FLUME-1482 for HTTP-based monitoring, which is checked in on trunk.

        Show
        Mike Percy added a comment - Well we already have FLUME-1482 for HTTP-based monitoring, which is checked in on trunk.
        Hide
        Denny Ye added a comment -

        HTTP monitoring component is just an example. We have our HTTP checking and collection application. Flume is one of agent to support HTTP service in application cluster. Thus, another service should be manage except regular Source and so on.
        LifecycleAware is perfect choice for us to extend function of Flume. Therefore, no any place to manage such extensions. What's your suggestion?

        Show
        Denny Ye added a comment - HTTP monitoring component is just an example. We have our HTTP checking and collection application. Flume is one of agent to support HTTP service in application cluster. Thus, another service should be manage except regular Source and so on. LifecycleAware is perfect choice for us to extend function of Flume. Therefore, no any place to manage such extensions. What's your suggestion?
        Hide
        Mike Percy added a comment -

        I apologize, but I'm still confused about what the problem is that you want to solve.

        Show
        Mike Percy added a comment - I apologize, but I'm still confused about what the problem is that you want to solve.

          People

          • Assignee:
            Unassigned
            Reporter:
            Denny Ye
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development