Pretty much done, all tests pass. It breaks back compat, but frankly, straddling doesn't seem worth the effort here. Or even very possible. You can't really give new methods to use for the deprecated ones, and deprecating by class would be a real nuisance as we would lose class names I'd rather keep. We have no back compat policy, and I think its worth just pushing this to the new API.
I was also thinking about breaking back compat with changing the Highlighter to use the SpanScorer, so doing it all in one shot would be nice. The overall migration should be fairly simple once you understand the new TokenFilter API. I'll handle it for Solr.
Still needs either its own changes file to explain or could go in the contrib common changes file.
There is a change to the MemoryIndex to get around issues with the new/old API and CachingTokenFilters.
Ill have to see how the new TokenFilter API improvements issue works out before doing a final patch for this.