Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-7276

PutFTP create alot of received trafic when tx small files

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.11.4
    • Fix Version/s: None
    • Component/s: Extensions
    • Labels:
    • Environment:
      NIFI: Ubuntu 18.04. NIFI 1.11.4
      FTP: Ubuntu 18.04 with vsftpd

      Description

      I don't know if this is a bug, but the result of my investigation is that it seems like the PutFTP receive between 1,6 - 2,7MB of data per file it transmit. For me it seems like a lot of RX data.

      I'm using nload and nethogs to monitor the network trafic.

      I have a lot (100.000+) of small files (<1MB), which I can't combine.

      When starting the PutFTP process the nifi server starts receiving data from the ftp server and it continues with with an avg. of 50 Mbit/s, constantly.

      To increasing the throughput I've set the run duration to 2sec. and playing with the Batch size. With a batch size of 100 and over 5 min. I can see at the ftp server that nifi is logging in 5 times and start a vsftpd process. within those 5 min. nifi has tx 700 files and 65.32 MB, but the ftp server has sent 1544MB data back to nifi.

      With a batch size of 1000. It took nifi around 8 minutes for transferring 1000 files with a total sum of 64MB, but the ftp server has this time TX 2700MB data back to nifi.

      I tried with larger files and a batch size at 100. NIFI sent 249 files with at total sum of 32 GB of data within 3½ min. and this time the ftp server sent 400MB data back to nifi.

      It seems like the PutFTP receive between 1,6 - 2,7MB of data per file it transmit. For me it seems like alot of RX data.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jmkofoed Jens M Kofoed

              Dates

              • Created:
                Updated:

                Issue deployment