Major Description
After the changes in SAMZA-111, the Samza's container's slowest operations seem to be log-related. Most of our CPU time is spent in grizzled's SL4J logging. Take a look at SAMZA-111.0.png for a screenshot of the CPU sample, and you can see it's nearly all trace and debug statements that are taking the time.
Also, in SAMZA-111.0.png, the window (15.9%), commit (13%), and send (10.4%) are all logging as well. I just couldn't fit it in the screenshot. This combines for 39.3% of the total CPU time in the main process loop. On top of this, window, commit, and send are entirely no-ops.
We should improve the log performance so that it's not having such a bad impact on throughput. After this ticket is resolved, we should open a new ticket with the next slowest item in Samza container, and make it faster.