[HTTPCLIENT-2006] HttpClient 5.0-beta5 gets 403 error for URL, works in HttpClient 4/Curl/Wget etc

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 5.0 Beta5
• Component/s: HttpClient (classic)
• Environment: Java 12/MacOS 10.14.6
• Assignee: Unassigned
• Reporter: Andrew Parks
• Created: 31/Jul/19 12:12
• Updated: 31/Jul/19 14:18
• Resolved: 31/Jul/19 14:18

Description

When testing an upgrade from HttpClient 4.5.9 to HttpClient 5.0-beta5, I tried the following code on a large number of URLs:

System.out.println(Request.Get("https://www.popsugar.com").execute().returnContent());

It worked fine for other sites, but for some reason, the PopSugar web site returns a 403 error. The same web site does not have a problem when HttpClient 4.5.9 is used to do exactly the same thing. There is also no problem when tested with curl, wget, or a regular browser. Changing user agents does not make a difference.

I have verified that the PopSugar web site supports HTTP/2. Perhaps there is a protocol incompatibility.