Create an open source system to gather ATS request / response logs.
1. Must be built from all open source components.
2. Must be able to ingest 1MM records a second on a reasonable amount of hardware and be horizontally scalable to add additional capacity.
3. Must allow storage of raw logs for configurable period of time between 30-90 days.
4. Must be able to process raw logs from ATS and split based service by assessing configuration of service.
5. Must be able to split logs into files of no more than 2 hours or 500k logs whichever occurs first. These should be starter configs and should be configurable on a per service basis.
6. Log files must be compressed to conserve space.
7. Log files should be accessible via a web server interface.
8. Log files should be accessible via an API request.