About the fix itself: If you can find a document or standard that describes a standard for URL transformation that is not supported by the Web Connector, or you can show that such a transformation works in IE and in Firefox, then we should modify the WebURL class to support this transformation, and others that are similar. I created the WebURL class specifically for the purpose of providing support for URL forms that were unsupported by the Java implementation of URI, which is no longer up-to-date as far as standards are concerned. So, if there was going to be a fix for this issue, I'd recommend that it be done there, and not in WebcrawlerConnector.
My understanding of how URL encoding was supposed to work was that a URL is encoded in links, NOT by the browser (or crawler). This is necessary because the browser does not typically understand the context within a URL correctly. Now, Microsoft modified that standard by supporting certain transformations within IE, and other browsers have copied those transformations. If there is sufficient support across browsers, we should go ahead and provide a similar feature in the web connector.
In order to check whether your transformation of full-width space qualifies as feature we should support, you would want to create a website locally (running under IIS probably), which has documents with names that include problematic characters such as full-width space. Then, also create a page that has links to these documents, in the form <a href="...">...</a>, where the full-width space character is NOT properly URL encoded but is exposed. Browse to the link page and click on the link. Does the browser load the expected document, or not? Which browsers work, and which do not? If it does seem to be supported, are there other characters that work the same, or not?