Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.11.4
-
None
-
NIFI: Ubuntu 18.04. NIFI 1.11.4
FTP: Ubuntu 18.04 with vsftpd
Description
I don't know if this is a bug, but the result of my investigation is that it seems like the PutFTP receive between 1,6 - 2,7MB of data per file it transmit. For me it seems like a lot of RX data.
I'm using nload and nethogs to monitor the network trafic.
I have a lot (100.000+) of small files (<1MB), which I can't combine.
When starting the PutFTP process the nifi server starts receiving data from the ftp server and it continues with with an avg. of 50 Mbit/s, constantly.
To increasing the throughput I've set the run duration to 2sec. and playing with the Batch size. With a batch size of 100 and over 5 min. I can see at the ftp server that nifi is logging in 5 times and start a vsftpd process. within those 5 min. nifi has tx 700 files and 65.32 MB, but the ftp server has sent 1544MB data back to nifi.
With a batch size of 1000. It took nifi around 8 minutes for transferring 1000 files with a total sum of 64MB, but the ftp server has this time TX 2700MB data back to nifi.
I tried with larger files and a batch size at 100. NIFI sent 249 files with at total sum of 32 GB of data within 3½ min. and this time the ftp server sent 400MB data back to nifi.
It seems like the PutFTP receive between 1,6 - 2,7MB of data per file it transmit. For me it seems like alot of RX data.