Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-1842

implement GetWithCamel and PutWithCamel processors, as selectable-endpoint connectors

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Extensions
    • Labels:
      None

      Description

      While the SpringContextProcessor (NIFI-1571) provides an excellent basis for running Camel, it still requires the user to program Camel. In light of the fact that Camel provides some 80+ protocol-aware endpoints (see the list at https://github.com/apache/camel/tree/master/components), at least half of which are probably useful to NiFi and not yet available as NiFi-native processors, I propose the following:

      Create processors named GetWithCamel and PutWithCamel, that allow:

      • configuration-time selection from a subset of Camel-supported endpoints
      • and then providing the additional config values required by the selected endpoint, probably in an XML file

      These processors will of course be based on SpringContextProcessor, but hardwired for Camel and with added configuration glue in the UI to allow selecting a Camel endpoint without having to write Camel or Spring code. This won't be possible for all the endpoints, but will work for many useful ones.

      DESIGN ISSUES to be resolved (Thanks to Joe Witt, Puspendu Banerjee, Oleg Zhurakousky, for raising these items):

      • We want to add this feature without requiring that big blocks of Camel or Spring dependency code get sucked in to NiFi deployments that don't require them. Appropriate bundle isolation (at NiFi and possibly Camel levels), plus use of dynamic class loaders, are clearly part of the solution, but this strategy needs to be elaborated. The upcoming NiFi Extension Registry may also be part of the solution.
      • We want to avoid run-time, or even configuration-time, dependency resolution. While maven is a widely acknowledged utility for dependency resolution, it is not allowable in all environments. Resolving all dependencies into a metadata form at build time is preferable, and I believe achievable. A dependency resolver capable of downloading dependencies from maven repositories at config time, if not present in classpath, may still be useful as an option in some environments. The upcoming NiFi Extension Registry may also be part of the solution, if trusted in deployment environments.
      • The provenance-reporting feature of NiFi must be supported. How to integrate this with Camel end-points needs design work. It may need to be different for different end-points.
      • Effective code review will require patience, since Spring and Camel expertise may be less available in the community. Maintainability of the new processors must also be considered, given this limitation.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                mattf Matthew Foley
              • Votes:
                6 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated: