I have a few comments about the patch. Overall it looks very good.
If the old counter API is going to be the long term fix then perhaps we should not mark is as deprecated any more. If not then can we file a new JIRA to update the new Counters to be as space efficient as the old ones.
This is a minor performance improvement, but in many of the HistoryEvents there is a datum, that has had most of its fields replicated inside the event itself. I guess this is because they are in slightly different formats now. But we still create an instance of datum in the constructor, and always populate its fields inside getDatum. These event objects are imutable, outside of the setDatum method that is only to be used when de-serializing the event. getDatum, however, tends to be called repeatedly to pull out individual fields from the datum. I would prefer to see the datum start out as null, and only have its fields set if it is not null, inside getDatum.
Why was TokenCache.java modified at all? It does not seem to be related to this JIRA.
You added a TODO in CompletedTask.java and CompletedJob.java
Did you make sure yet? if so please delete the TODO.
Also good catch on TestHsWebServicesTasks.java, TestAMWebServicesAttempts.java, TestHsWebServicesTasks.java and the others. I also don't think we need any more tests because, all we are doing is reducing memory usage, which is very hard to write a unit test for.
Inside JobHistoryEventHandler.java you added in
is this TODO still needed?
Like I said before, overall the patch looks really good and I only have some minor comments, thanks for doing this.