RE: implementing custom MergePolicy
Let me describe in detail on problems of implementing a custom MergePolicy:
1) In IndexWriter code, such methods on MergePolicy is called, e.g. findMergesForOptimize. I believe that is the contract for implementing your own MergePolicy. However, it is "hidden" by the javadoc in terms of documentation, and furthermore, it is hidden because these methods are package protected. So to implement your own MergePolicy, you have to resort back to sneaking the class into the package.
2) Not only seg/getUseCompoundFile is no longer applicable if LogMergePolicy is not used, also popular methods such as set/getMergeFactor etc. are only applicable to LogMergePolicy. (Just to clarify, useCompoundFile is a package-level protected method on the base MergePolicy class, so my guess is that set/getCompoundFile should be applicable to all implementations of MergePolicy.
This brings up another issue about the practice of having to "sneak" classes into a package. We are looking at making our Lucene code, OSGI compliant, and this becomes an issue because we cannot have multiple "bundles" exporting the same package. Which means, I would have to repackage lucene to include my classes that I have snuck into some lucene packages. I would like to use a standard distribution of a lucene jar (as suggested/echoed by some luceners).