Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6831

FileStreamSink is very slow

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 1.1.0
    • Fix Version/s: None
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      Hi Team,

       

      I am very new in kafka. My project requirement is fetch data from source location and place it in other other location (consumer location). I am using FileStreamSink class to perform above action.

      I am using Linux machine having memory of 32 GB. 

      When i start FIleStreamSink , It is syncing to consumer location very very slowly. Not sure why it is taking 2000 message at a time and then sync it. After that it wait for few second then sync again. This waiting time increases per run .

       

      I am processing 600K message but it took 1 hrs to process only 60K message.

       

      Below are my config details : 

       

      connect-file-sink.property

      Name = local-file

      Connector.class = FileStreamSource

      task.max=20

      file=/d/d1/kafka/destination/outfile.txt

      topic=abc_partion_20

      connect-file-source.property

      Name = local-file

      Connector.class = FileStreamSource

      task.max=20

      file=/d/d1/kafka/source/infile.txt

      topic=abc_partion_20

       

      Can you please help ?

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              vrmprabhat Prabhat Verma
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: