Suggested fix:

    try {
      scorer.setUsePayloads(request.getParams().getFieldBool(fieldName, HighlightParams.PAYLOADS,
          request.getSearcher().getLeafReader().fields().terms(fieldName).hasPayloads()));
      // It'd be nice to know if payloads are on the tokenStream but the presence of the attribute isn't a good
      // indicator.
    } catch (IOException e) {
      throw new RuntimeException(e);
    }

I'm going to try this now with Solr's tests, then post a patch.

This was discovered via a commenter here: https://issues.apache.org/jira/browse/SOLR-5855?focusedCommentId=14578437&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14578437 (the purpose of that issue is unrelated to the discovery in the stack traces posted)

Here's a patch; a little better than the "suggested fix": the Terms returned could be null, and if an exception somehow gets thrown then we can log it without re-throwing.

I did a little performance testing on a project I have. It seems this perf bug is most pronounced if you attempt to highlight on a ton of fields (e.g. via hl.fl=*), and if there are a lot of Lucene segments. And furthermore if you don't have a lot of text to highlight per field then the overhead here is proportionally higher to the overall task.

Precommit is happy and the tests pass. It'd be nice to get this into 5.2.1 but would like to see a +1 from someone. What do you think Shalin Shekhar Mangar (you're the RM I believe).

As a side note... I'm wondering if SlowCompositeReaderWrapper ought to cache FieldInfos too; maybe lazyily.

Looks good to me. I don't know if you're online but if not, I'll commit your patch in a couple of hours and create the RC.

Took me a while to get a build environment up, but results look very promising. My very unscientific benchmark shows that the highlighter may now be even a bit faster than in Solr 4.10.2 at least in some cases. Thanks for the great work, David Smiley!

Commit 1684665 from David Smiley in branch 'dev/trunk'
[ https://svn.apache.org/r1684665 ]

SOLR-7655: Speed up DefaultSolrHighlighter's check for the existence of payloads

Commit 1684667 from David Smiley in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1684667 ]

SOLR-7655: Speed up DefaultSolrHighlighter's check for the existence of payloads.

Commit 1684668 from David Smiley in branch 'dev/branches/lucene_solr_5_2'
[ https://svn.apache.org/r1684668 ]

SOLR-7655: Speed up DefaultSolrHighlighter's check for the existence of payloads.

Shalin Shekhar Mangar I didn't see you online and I saw it wasn't committed yet so I did so now.