Uploaded image for project: 'Community Development'
  1. Community Development
  2. COMDEV-360

RocketMQ Connect Hive

    XMLWordPrintableJSON

    Details

      Description

      Content

      The Hive sink connector allows you to export data from Apache RocketMQ topics to HDFS files in a variety of formats and integrates with Hive to make data immediately available for querying with HiveQL. The connector periodically polls data from RocketMQ and writes them to HDFS.

      The data from each RocketMQ topic is partitioned by the provided partitioner and divided into chunks. Each chunk of data is represented as an HDFS file with topic, queueName, start and end offsets of this data chunk in the filename.

      So, in this project, you need to implement a Hive sink connector based on OpenMessaging connect API, and run it on RocketMQ connect runtime.

      You should learn before applying for this topic
      Hive/Apache RocketMQ/Apache RocketMQ Connect/ OpenMessaging Connect API

      Mentor

      chenguangsheng@apache.orgvongosling@apache.org

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              duheng duheng
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: