I can't really explain what was happening, but for a time, many valid links would throw an exception. Then it just stopped. I think we don't really know what is going on in the code. LIke, what really causes an exception to get thrown? I don't see the possibility for an infinite loop.
I for one still don't trust that links that throw an exception are really problematic, and think that having one such link shouldn't stop parsing. I am guessing that failed links aren't recorded or generally reviewed, so I see this as a place that parsing and crawling could fail and it would be pretty hard to track down. Just seems a little too unforgiving.