The problem is that scorers are hard to track: scoring usually happens by calling Scorer.score(Collector), which itself calls Collector.setScorer(Scorer). Since the asserting scorer delegates to the wrapped one, the asserting scorer gets lost, this is why Collector.setScorer tries to get it back by using a weak hash map.
I'm not totally happy with it either and would really like to make Scorer.score(Collector) use methods from the asserting scorer directly. We can't rely on Scorer.score(Collector)'s default implementation since it relies on Scorer.nextDoc and some scorers such as BooleanScorer don't implement this method.
Could we alternatively use VirtualMethod to detect if score(Collector)/score(Collector,int,int) are overridden in the underlying scorer? If they aren't, then its safe for AssertingScorer to use its own implementation (possibly with more checks).