I don't see why adding generics would complicate or limit the API. To me it would make it simpler and nicer (not a big change in terms of api itself though).
Attaching a patch with my thoughts to make more concrete what I had in mind, regardless of whether it will be integrated or not.
It's backwards compatible (even though the class is marked experimental): we have an abstract postings highlighter class that does most of the work and returns arbitrary objects (uses generics in order to do so). The PostingsHighlighter is its natural extension that returns String snippets.
I updated Mike's new test according to my changes. It should make it easier to understand what's needed to work with arbitrary objects in terms of code using this approach.
I find it more explicit that if you want to extend the abstract one you have to declare what type the formatter is supposed to return, which makes it more explicit and avoids any cast.
Limitations with this approach:
1) as mentioned before (to me it's more of a benefit) there cannot be heterogeneous types returned by the same highlighter instance.
2) generics don't play well with arrays, thus all the highlight methods that returned arrays are still in the subclass that returns string snippets to keep backwards compatibility. Moving them to the base class would most likely require to return List<FormattedPassage> instead (not backward compatible).
I haven't updated the javadoc yet, but if you like my approach I can go ahead with it.
I would love to hear what you guys think about it. Generics can be scary... but useful sometimes too