Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-16754

Camel Kafka HDFS sink connector

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Invalid
    • 3.9.0, 3.10.0
    • None
    • camel-hdfs, camel-kafka
    • None
    • Novice

    Description

      Hello, 

      I'm trying to connect kafka and hdfs to store data. I set it up and it works correctly, but the problem arises when I save the kafka messages in hdfs as a file is created for each message. I would like to create a file containing multiple messages, but I can't solve this problem. I've change the value of 

      camel.sink.endpoint.splitStrategy=BYTES:1000000
      
      camel.sink.endpoint.splitStrategy=MESSAGES:10
      

      But when I view the files in the hdfs folder, I see one file for each message (image adjunted).
      The full configuration of the connector is the next:

      name=CamelHdfsSinkConnector
      connector.class=org.apache.camel.kafkaconnector.hdfs.CamelHdfsSinkConnector
      tasks.max=1
      
      # use the kafka converters that better suit your needs, these are just defaults:
      key.converter=org.apache.kafka.connect.storage.StringConverter
      value.converter=org.apache.kafka.connect.storage.StringConverter
      #key.converter=org.apache.kafka.connect.json.JsonConverter
      #value.converter=org.apache.kafka.connect.json.JsonConverter
      # comma separated topics to get messages from
      topics=modbus-office-topic
      # mandatory properties (for a complete properties list see the connector documentation):
      # HDFS host to use
      camel.sink.path.hostName=namenode
      camel.sink.path.port=9000
      camel.sink.endpoint.splitStrategy=BYTES:10000000
      # The directory path to use
      camel.sink.path.path=Example_folder
      

      I am currently running hadoop version 3.1.2, I have my doubts that this is the problem, and I don't know if the problem is with the connector, the hadoop version or the connection configuration.
      Thanks for your time

      Attachments

        1. Screenshot_1.png
          167 kB
          Fernando

        Activity

          People

            Unassigned Unassigned
            fdorado Fernando
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 0.5h
                0.5h
                Remaining:
                Remaining Estimate - 0.5h
                0.5h
                Logged:
                Time Spent - Not Specified
                Not Specified