Uploaded image for project: 'Falcon'
  1. Falcon
  2. FALCON-2299

Adding custom engine in Apache Falcon

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.9
    • Fix Version/s: 0.9
    • Component/s: general
    • Labels:
    • Flags:
      Important

      Description

      Falcon currently supports 4 engines - oozie, pig, hive, spark. Is it possible to configure Falcon to include another engine apart from the ones already provided ?
      We might be able to contribute to the Falcon project source code in order to support custom engine. But is there any other way using which we can plug custom engine to an existing Falcon build instead of creating our own Falcon build which supports our engine ?
      Has falcon opened its api for letting developers integrate their own tools/engines ?

        Activity

        Hide
        pallavi.rao Pallavi Rao added a comment -

        Aniket Rajendra More, Falcon's design is quite well-structured in terms ease of adding a new execution engine. However, it does not follow a plugin/extension model, where users can just plug a new engine in. You will have to make changes to the core components of Falcon. If you have a good use case for supporting a generic engine, you can branch off the code and work on the same. You will get the community's input/help along the way.

        Also, just wanted to point out that pig, hive and spark are also internally translated to corresponding Oozie actions. So, when you say an execution engine, I'm not sure if you are planning add another type of action supported by Oozie (going by your example) or an engine that is parallel to Oozie. Either way, the design is extensible.

        Show
        pallavi.rao Pallavi Rao added a comment - Aniket Rajendra More , Falcon's design is quite well-structured in terms ease of adding a new execution engine. However, it does not follow a plugin/extension model, where users can just plug a new engine in. You will have to make changes to the core components of Falcon. If you have a good use case for supporting a generic engine, you can branch off the code and work on the same. You will get the community's input/help along the way. Also, just wanted to point out that pig, hive and spark are also internally translated to corresponding Oozie actions. So, when you say an execution engine, I'm not sure if you are planning add another type of action supported by Oozie (going by your example) or an engine that is parallel to Oozie. Either way, the design is extensible.
        Hide
        Aniketmo Aniket Rajendra More added a comment -

        Thanks for the explanation above Pallavi Rao
        As you have mentioned above, the design is extensible for adding an engine that is parallel to oozie. Is it externally configurable or do we need code changes in core components of Falcon?
        Can you also provide some reference documents or links.

        Show
        Aniketmo Aniket Rajendra More added a comment - Thanks for the explanation above Pallavi Rao As you have mentioned above, the design is extensible for adding an engine that is parallel to oozie. Is it externally configurable or do we need code changes in core components of Falcon? Can you also provide some reference documents or links.

          People

          • Assignee:
            Unassigned
            Reporter:
            Aniketmo Aniket Rajendra More
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development