[HTTPCLIENT-1149] EntityUtils.toString should detect Byte order mark (BOM) and remove it if present

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 4.1.2
• Component/s: HttpClient (classic)
• Labels: BOM EntityUtils
• Environment: Windows
• Assignee: Unassigned
• Reporter: Ian Beaumont
• Created: 01/Dec/11 15:41
• Updated: 05/Oct/13 23:41
• Resolved: 01/Dec/11 21:37

Description

The Byte order mark at the start of the input stream should be detected and removed by EntityUtils.toString, otherwise strange unwanted characters are left at the start.
This link lists possible Byte order markings http://en.wikipedia.org/wiki/Byte_order_mark
I'm not sure if EntityUtils.toString using the BOM to try to detect the encoding, but if it doesn't then it should.

Example URL that is causing this issue is mircosoft virtual earth WSDL file:
HttpClient httpclient = new DefaultHttpClient();
HttpGet httpget = new HttpGet("http://dev.virtualearth.net/webservices/v1/searchservice/searchservice.svc?wsdl");
HttpResponse response = httpclient.execute(httpget);
HttpEntity entity = response.getEntity();
String textContents = EntityUtils.toString(entity);