With a heavy enough write load, it appears that the E2E agent WAL will get into a state where data just gets constantly shuffled around between the various directories / states (e.g. writing, logged, sending, sent). When this happens, the WAL directories grow indefinitely until the disk is exhausted, no matter how much data caused the problem.
- Use the supplied config (or something similar).
- Write to the agent source at a rate of > 1MB/s for a short burst (using something like the provided generator below).
- Note that data is delivered to the collectorSink but the agent WAL manager constantly grows the data.
This looks and smells just like
FLUME-430. I haven't yet examined the WAL or destination data for duplicates / missing events.