I haven't tried it in production yet. Nonetheless, I have included this change in our benchmarks configuration:
- without the change, the agent needs more or less 30 minutes to send one 490MB file with the file tailer adaptor.
- with the change, it only needs 45 seconds
I don't really understand what you mean by "rate limiting". The code already offers some protection
to ensure that one (big) file to tail may not prevent other smaller files to be tailed (see chukwaAgent.fileTailingAdaptor.maxReadSize).
What is more, I think my change is correct if you consider the comment at the beginning of the class:
- For now, it tries each file in succession. If it gets through every file
- within two seconds, and no more data remain, it will sleep.
- If there was still data available in any file, the adaptor will loop again.