Patch does not handle all files well (few tests fail). Apparently, the .del file should not be rolled into the .cfs.
Right, .del files never appear inside a CFS.
SegmentMerger.createCompoundFile does this by default, however it's only called from code that ensures no deletions exist. Would have been nice if this method documented it .
Please add comments to this! It's non-obvious
Also, I think *.s<num> should not be rolled into .cfs (those are the separate norms files). I don't know how to create such files in the first place (thought they're of old format, but 3.1 indexes have them also), and TestBackCompat fails.
Right, these too only live outside a CFS. You create them by opening a writable IndexReader (I know: confusing!) and calling setNorm, then closing it. They are not only for old indices... 4.0 creates them too.
Is there a way to identify those files? Is it safe to check if the file extension starts w/ IndexFileNames.SEPARATE_NORMS_EXTENSION? Feels hacky to me.
Hackish though it seems (I agree) I think that's the only way? SegmentInfo.hasSeparateNorms is equally hacky...
Another thing, I think in order to avoid shared doc stores (and whatever other old-format) stuff, since it's only an optimization, that the code should copy into CFS only if the segment version is on or after 3.1 (that is StringHelper.getVersionComparator().compare(info.getVersion, "3.1") >= 0).
Shared doc stores, yes, but the separate del docs / norms are produced by all versions.
More generally: does addIndexes properly refuse to import a too-old index? We should throw IndexFormatTooOldExc in this case? (And, maybe also IndexFormatTooNewExc?).