Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-6786

Adding KafkaConsumerHook, KafkaProducerHook, and KafkaSensor

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: In Progress
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.10.9
    • Fix Version/s: None
    • Component/s: contrib, hooks
    • Labels:
      None

      Description

      Add the KafkaProducerHook.
      Add the KafkaConsumerHook.
      Add the KafkaSensor which listens to messages with a specific topic.
      Related Issue:
      #1311 (Pre-dates Jira Migration)

      Reminder to contributors:

      You must add an Apache License header to all new files
      Please squash your commits when possible and follow the 7 rules of good Git commits
      I am new to the community, I am not sure the files are at the right place or missing anything.

      The sensor could be used as the first node of a dag where the second node can be a TriggerDagRunOperator. The messages are polled in a batch and the dag runs are dynamically generated.

      Thanks!

      Note, as per denied PR #1415, it is important to mention these integrations are not suitable for low-latency/high-throughput/streaming. For reference, #1415 (comment).

      Co-authored-by: Dan Ferguson dferguson992@gmail.com
      Co-authored-by: YuanfΞi Zhu

        Attachments

          Activity

            People

            • Assignee:
              dferguson992 Daniel Ferguson
              Reporter:
              dferguson992 Daniel Ferguson
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: