Here's a patch; a little better than the "suggested fix": the Terms returned could be null, and if an exception somehow gets thrown then we can log it without re-throwing.
I did a little performance testing on a project I have. It seems this perf bug is most pronounced if you attempt to highlight on a ton of fields (e.g. via hl.fl=*), and if there are a lot of Lucene segments. And furthermore if you don't have a lot of text to highlight per field then the overhead here is proportionally higher to the overall task.
Precommit is happy and the tests pass. It'd be nice to get this into 5.2.1 but would like to see a +1 from someone. What do you think Shalin Shekhar Mangar (you're the RM I believe).
As a side note... I'm wondering if SlowCompositeReaderWrapper ought to cache FieldInfos too; maybe lazyily.