[NUTCH-3001] protocol-selenium requires Content-Type header

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.20
• Assignee: Unassigned
• Reporter: Tim Allison
• Created: 13/Sep/23 13:58
• Updated: 13/Mar/24 14:51
• Resolved: 13/Sep/23 18:58

Description

It looks like the selenium protocol requires that there be a content-type header.

The logic seems to be: If the content type is html or xhtml, use selenium, otherwise just grab the bytes.

However, with the current logic, if the content-type is null, nothing is pulled.

My guess is that the logic should be : if the content type is not null and equals html or xhtml use selenium, otherwise grab the bytes.

Right?

      String contentType = getHeader(Response.CONTENT_TYPE);

      // handle with Selenium only if content type in HTML or XHTML
      if (contentType != null) {
         if (contentType.contains("text/html")
            || contentType.contains("application/xhtml")) {
               readPlainContent(url);
         } else {
...

Issue Links

• links to: GitHub Pull Request #774