Someday maybe I'll convince you to donate this "schema" layer on top of Lucene
It's not generic enough to be of use for every user of Lucene, and it doesn't aim to be such.
Solr has its own schema approach, and it has its merits and downfalls compared to mine. That's what is nice, we're able to use the same library in differing ways, and it doesn't force its sense of 'best practices' on us.
There's no forcing going on, here. Even had we added the bit into the
index, there's still no "forcing". We're not preventing advanced uses
of Lucene by providing strong Numeric* support in Lucene. Simple
things should be simple; complex things should be possible...
But I hope there are SOME named classes in there and not all static factory methods returning anonymous untyped impls.
SOME of them aren't static :-D
We shouldn't weaken trie's integration to core just because others have private implementations.
You shouldn't integrate into core something that is not core functionality. Think microkernels.
It's strange seeing you drive CSFs, custom indexing chains, pluggability everywhere on one side, and trying to add some weird custom properties into index that are tightly interwoven with only one of possible numeric implementations on the other side.
I agree: if Lucene had all extension points that'd make it possible
for a good integration of Numeric* without being in "core", we should
use that. But we're just not there yet. We want to get there, and we
will, but we can't hold up progress just because we think someday
we'll get there. That's like saying we can't improve the terms dict
format because it's not pluggable yet.
Design for today.
And spend two years deprecating and supporting today's designs after you get a better thing tomorrow. Back-compat Lucene-style and agile design aren't something that marries well.
donating something to Lucene means casting it in concrete.
We can't let fear of back-compat prevent us from making progress.
IndexReader/Writer pair is a good example of what I'm arguing against. A dusty closet of microfeatures that are tightly interwoven into a complex hard-to-maintain mess with zillions of (possibly broken) control paths - remember mutable deletes/norms+clone/reopen permutations? It could be avoided if IR/W were kept to the bare minimum (which most people are going to use), and more advanced features were built on top of it, not in the same place.
Sure, our approach today isn't perfect ("progress not perfection").
There are always improvements to be done. If you see concrete steps
to simplify the current approach without losing functionality, please
post a patch. I too would love to see such simplifications...
NRT seems to tread the same path, and I'm not sure it's going to win that much turnaround time after newly-introduced per-segment collection.
I agree, per-segment collection was the bulk of the gains needed for
NRT. This was a big change and a huge step forward in simple reopen
But, not having to write & read deletes to disk, not commit (fsync)
from writer in order to see those changes in reader should also give
us decent gains. fsync is surprisingly and intermittently costly.
And this integration lets us take it a step further with
where recently created segments can remain in RAM and be shared with
If you have good simplifications/improvements on the approach here,
please post them.
Some time ago I finished a first version of IR plugins, and enjoy pretty low reopen times (field/facet/filter cache warmups included). (Yes, I'm going to open an issue for plugins once they stabilize enough)
I'm confused: I thought that effort was to make SegmentReader's
components fully pluggable? (Not to actually change what components
SegmentReader is creating). EG does this modularization alter the
approach to NRT? I thought they were orthogonal.
If we add some generic storable flags for Lucene fields, this is cool (probably), NumericField can then capitalize on it, as well as users writing their own NNNFields.
+1 Wanna make a patch?
No, I'd like to continue IR cleanup and play with positionIncrement companion value that could enable true multiword synonyms.
Well I'm looking forward to seeing your approach on these two!