Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Noticed that during high ingest load, there's a bit of downtime between iterations as, after the batch size is reached, the tserver records the progress it made, and it waits for the master to re-queue the file.
This is a bit silly as the tserver could just continue to replicate from where it left off to reduce the latency in waiting to replicate the file. The remote side will still buffer, so we shouldn't be oversaturating the remote.