Thanks everyone for reviewing!
New patch which does the following -
- Refactored the DirectoryFileStream and LocalFsFileStream to reuse code. The throttling logic is folded in DirectoryFileStream instead of having a separate implementation
- The init pause is needed for the future pause timings to work correctly. Check TestRateLimiter.testPause() which does the same thing. Test fails when removing the init in testPause() method. We should fix this in another Jira I guess.
- Test case and some small refactoring in TestReplicationHandler
About reserving index commit points, currently we extend it by 10s for every 5 reads of 1MB ( default ) packet sizes. If one throttles a lot this could trip like Ramkumar pointed out.
So we could take two approaches -
1. Instead of the hardcoded 10 seconds look at the throttle speed and estimate how much time should the reserve point be extended for.
2. Add functionality to the RateLimiter that actually returns the time it needs to pause before actually pausing. We could split the current pause method into 3 methods -
preparePause()- would return the pauseNS value.
pause(int pauseNS) - Pause for the specified time duration.
pause() - calls preparePause() and then pause(pauseNs) . This makes sure that we are not breaking the public api.
We could then take the time it needs to pause for and add it to the default 10 seconds.