I find the pluggable replace/prevent/append policy idea appealing, but I have not yet found a great way to plug it into the UpdateHandler. Any approach other than sub-classing DirectUpdateHandler2 appears to lead to tying an IndexWriter to UpdateHandler. There is a connection now, UpdateHandler has a method to create a main IndexWriter, but further tying seems wrong without a stronger reason. That point is arguable, but in the end, sub-classing results in simpler code in any case. The trade off is that now you have a PreventDupesDirectUpdateHandler that extends DirectUpdateHandler2. This would have to be used in combination with the SignatureUpdateProcessor if you want to prevent dupes from entering the index. Other use cases (other than overwriting) would require another UpdateHandler. Less than ideal in both cases (subclassing, pluggable interface/class).
Both approaches lead to less than ideal solutions beyond that as well . Because many docs that have been added to Solr might not yet be visible to an IndexReader, you have to keep a pending commit set of docs to check against. This list should be resilient against AddDoc, DelDocWquery, AddDoc, Commit. You'd essentially have to keep a mini index around to search against to accomplish this, due to delete by query. The other options are to either auto-commit sans a user commit before a delete, or just say we don't support that use case when using that UpdateHandler. None of it is very pretty.
Another option is to do things with an UpdateProcessor. This is the most elegant solution really, but it requires putting big,coarse syncs around the more precise syncs in DirectUpdateHandler2. That may not be a huge deal, I am not sure. The previous two options allow you to maintain similar syncs as to what is already there. Beyond that, the UpdateProcessor approach still has the delete by query issues.
Maybe we just do overwrite dupe for now? It has none of these issues. I am open to whatever path you guys want. The other use cases do have their place - we will just have to compromise some to get there. Or maybe there are other suggestions?
Another point that was brought up is whether or not to delete any docs that match the update docs uniqueField id term, but not its similarity/update term. At the current moment, IMO, we shouldn't. You are choosing to use the updateTerm to do updates rather then the unique term. This allows you to have duplicate signatures but also uniqueField Ids for other operations (say delete). Also, if you already have a unique field that your using, it may be desirable to do dupe detection with a different field. There is always the option of setting the signature field to the uniqueField term. In the end, your call, I'll add it if we want it.
As far as search time dupe collapsing, I think I could see a search component that takes the page numbers to collapse (start, end) and does dupe elimination on that range at query time. It wouldn't be very fast, and I'm not sure how useful page at a time collapsing is, but it would be fairly easy to do. Not sure that it fits into this issue, but certainly could share some of its classes.