Details
-
Type:
Improvement
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 6.0
-
Component/s: None
-
Labels:None
-
Lucene Fields:New
Description
DimensionalValues seems to be better across the board (indexing time, indexing size, search-speed, search-time heap required) than NumericField, at least in my testing so far.
I think for 6.0 we should move IntField, LongField, FloatField, DoubleField and NumericRangeQuery to backward-codecs, and rename with Legacy prefix?
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
+1 I was looking forward to this 
Can you please link to the benchmarks you did?
And I'm curious why some uses, say BBoxStrategy to pick one example, still use the Legacy version; why not the latest and greatest BKD? If it's just for precisionStep – by all means just toss the precisionStep configuration. It won't apply any more.
Can you please link to the benchmarks you did?
The results are a bit scattered ... I need to re-run and I think get a
blog post out describing all of this, at some point ... but in the
meantime, here are the benchmmarks:
LUCENE-6901 has the most recent index-time benchmarks, and LUCENE-6891
has the most recent search-time benchmarks and LUCENE-6881 has the
NumericField baseline.
The sources for these benchmarks are all in luceneutil.
And I'm curious why some uses, say BBoxStrategy to pick one example, still use the Legacy version;
Because this is the simplest cutover I could do for now. For
BBoxStrategy, it would be a major change (you must reindex) to
switch from postings to DimensionalValues ... I think we should
switch over consumers of LegacyNumeric* in follow-on
issues?
Because this is the simplest cutover I could do for now. For BBoxStrategy, it would be a major change (you must reindex) to switch from postings to DimensionalValues
Since this is for 6.0, a major release, so this is acceptable.
I think we should switch over consumers of LegacyNumeric* in follow-on issues?
Okay, if that's how you want to do it. But I figured the effort (albeit not "large") in using LegacyNumeric could have instead been directed on using DimensionalValues, and in the end there will be less effort to move fully over. For example adding dependency on backwards-codecs only to then take it away in another issue. Your call. Maybe I'm misjudging what's involved in switching to DimensionalValues but I hope it's easy.
Another patch, I think it's ready.
I fixed nocommits, added some limits on numBytes (16) and numDims (8), and fixed ant precommit, which was annoyingly not easy ... I could not get @link out to backward-codecs to render correctly into javadocs ... so I had to cutover to @code for links outside of core.
I could not get @link out to backward-codecs to render correctly into javadocs ... so I had to cutover to @code for links outside of core.
In modules you have to add <link/> elements inside <links/> of the <invoke-XXX-javadoc/> Ant element (should work with any of them). Example from benchmark module:
<invoke-module-javadoc>
<links>
<link href="../memory"/>
<link href="../highlighter"/>
<link href="../analyzers-common"/>
<link href="../queryparser"/>
<link href="../facet"/>
<link href="../spatial"/>
</links>
</invoke-module-javadoc>
If the javadoc target does not exist, you have to add it first.
Thanks Uwe Schindler, I did exactly that, however the javadocs would still point to core instead of the module, I suspect because the same package name appears in both?
I'm attaching a tiny patch based on trunk showing the issue. If you apply this patch, which just adds an @link to GeoHashUtils from queryparser (whose build.xml already has the javadocs link as you described) to sandbox then cd to lucene/queryparser and run ant javadocs and then open lucene/build/docs/queryparser/org/apache/lucene/queryparser/classic/QueryParserBase.html and look for the GeoHashUtils link, you should see it (incorrectly) pointing to core, not sandbox:
Here is the nocommit link: <a href="../../../../../../core/org/apache/lucene/util/GeoHashUtils.html?is-external=true" title="class or interface in org.apache.lucene.util"><code>GeoHashUtils</code></a>.</div>
Am I doing something wrong?
I think you meant that links from "core" module that point to legacy class won't work. This is something different! I have no idea!
I think you meant that links from "core" module that point to legacy class won't work.
It's not that: it's a link from queryparser module to sandbox module, and it fails because the javadocs incorrectly links the class from sandbox as if it were in core. But talking to Robert Muir, he seems to know what's happening here and basically we should not be putting the same package in multiple modules.
So ... one idea is to be less ambitious here, and instead leave NumericField in core, but renamed to LegacyX and deprecated. Then the links will be fine since no classes moved, they were simply renamed.
OK here's a new, less ambitious patch: it just renames the NumericX classes to LegacyNumericX, but leaves them in Lucene's core module. Javadocs linting, thankfully, now passes.
I also like the legacy query to stay in core. Because the codec module should not contain classes for public API usage. Its just a module to plug into your classpath and SPI loads codecs from it, nothing more. No user-facing APIs. In Java 9 Jigsaw's speak it should only export the ServiceProvider implementations but hide every implementation detail
As I see in your patch, no more build.xml changes. Cool. LGTM.
BTW, there was bug in your earlier patch already: You did not move the PNG files with the formulas from core/doc-files to the codec package, so Javadocs-Linter should have brought a missing image error, too. But that is no longer a problem, javadocs still look fine.
BTW, there was bug in your earlier patch already
Hmm this is maybe a bug in the linter ... I don't think it noticed the broken image link!
I think the fix should be an easy one. Just also look at <img src="..."> in addition to <a href="..."> and others. I can look into that.
Commit 1719562 from Michael McCandless in branch 'dev/trunk'
[ https://svn.apache.org/r1719562 ]
LUCENE-6917: rename/deprecate numeric classes in favor of dimensional values
Hey Mike, I noticed MIGRATE.txt isn't correct now that these classes aren't in backwards-codec module.
I was trying to run a single test case for Solr,
ant test -Dtestcase=SolrExampleXMLTest
I see some javadocs failures related to this commit:
compile-core:
[javac] Compiling 80 source files to /home/ishan/code/lucene-solr-svn/lucene/build/core/classes/java
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/DateTools.java:109: error: invalid use of @throws
[javac] * @throws ParseException if <code>dateString</code> is not in the
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/DateTools.java:123: error: invalid use of @throws
[javac] * @throws ParseException if <code>dateString</code> is not in the
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/DimensionalBinaryField.java:97: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name or value is null.
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/DimensionalDoubleField.java:81: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name or value is null.
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/DimensionalFloatField.java:81: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name or value is null.
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/DimensionalIntField.java:81: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name or value is null.
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/DimensionalLongField.java:81: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name or value is null.
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/Field.java:87: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if either the name or type
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/Field.java:106: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if either the name or type
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/Field.java:109: error: invalid use of @throws
[javac] * @throws NullPointerException if the reader is null
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/Field.java:138: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if either the name or type
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/Field.java:141: error: invalid use of @throws
[javac] * @throws NullPointerException if the tokenStream is null
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/Field.java:171: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name is null,
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/Field.java:173: error: invalid use of @throws
[javac] * @throws NullPointerException if the type is null
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/Field.java:189: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name is null,
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/Field.java:191: error: invalid use of @throws
[javac] * @throws NullPointerException if the type is null
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/Field.java:205: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name is null,
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/Field.java:207: error: invalid use of @throws
[javac] * @throws NullPointerException if the type is null
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/Field.java:228: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if either the name or value
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/Field.java:231: error: invalid use of @throws
[javac] * @throws NullPointerException if the type is null
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/Field.java:445: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if this field is not indexed,
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/FieldType.java:119: error: invalid use of @throws
[javac] * @throws IllegalStateException if this FieldType is frozen against
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/FieldType.java:142: error: invalid use of @throws
[javac] * @throws IllegalStateException if this FieldType is frozen against
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/FieldType.java:166: error: invalid use of @throws
[javac] * @throws IllegalStateException if this FieldType is frozen against
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/FieldType.java:190: error: invalid use of @throws
[javac] * @throws IllegalStateException if this FieldType is frozen against
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/FieldType.java:214: error: invalid use of @throws
[javac] * @throws IllegalStateException if this FieldType is frozen against
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/FieldType.java:238: error: invalid use of @throws
[javac] * @throws IllegalStateException if this FieldType is frozen against
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/FieldType.java:261: error: invalid use of @throws
[javac] * @throws IllegalStateException if this FieldType is frozen against
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/FieldType.java:284: error: invalid use of @throws
[javac] * @throws IllegalStateException if this FieldType is frozen against
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/FieldType.java:299: error: invalid use of @throws
[javac] * @throws IllegalStateException if this FieldType is frozen against
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/FieldType.java:323: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if precisionStep is less than 1.
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/FieldType.java:324: error: invalid use of @throws
[javac] * @throws IllegalStateException if this FieldType is frozen against
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/FieldType.java:455: error: invalid use of @throws
[javac] * @throws IllegalStateException if this FieldType is frozen against
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/LegacyDoubleField.java:149: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name is null.
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/LegacyDoubleField.java:162: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name or type is null, or
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/LegacyFloatField.java:151: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name is null.
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/LegacyFloatField.java:164: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name or type is null, or
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/LegacyIntField.java:151: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name is null.
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/LegacyIntField.java:164: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name or type is null, or
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/LegacyLongField.java:159: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name is null.
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/LegacyLongField.java:172: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name or type is null, or
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/document/SortedNumericDocValuesField.java:60: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if the field name is null
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/index/FieldInfos.java:184: error: invalid use of @throws
[javac] * @throws IllegalArgumentException if fieldNumber is negative
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/search/CachingCollector.java:377: error: invalid use of @throws
[javac] * @throws IllegalStateException
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/search/CachingCollector.java:380: error: invalid use of @throws
[javac] * @throws IllegalArgumentException
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/search/Weight.java:85: error: invalid use of @throws
[javac] * @throws IOException if an {@link IOException} occurs
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/search/Weight.java:114: error: invalid use of @throws
[javac] * @throws IOException if there is a low-level I/O error
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/search/Weight.java:131: error: invalid use of @throws
[javac] * @throws IOException if there is a low-level I/O error
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/search/spans/SpanWeight.java:83: error: invalid use of @throws
[javac] * @throws IOException on error
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/search/spans/SpanWeight.java:115: error: invalid use of @throws
[javac] * @throws IOException on error
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/search/spans/SpanWeight.java:145: error: invalid use of @throws
[javac] * @throws IOException on error
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/util/LegacyNumericUtils.java:194: error: invalid use of @throws
[javac] * @throws NumberFormatException if the supplied {@link BytesRef} is
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/util/LegacyNumericUtils.java:206: error: invalid use of @throws
[javac] * @throws NumberFormatException if the supplied {@link BytesRef} is
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/util/LegacyNumericUtils.java:220: error: invalid use of @throws
[javac] * @throws NumberFormatException if the supplied {@link BytesRef} is
[javac] ^
[javac] /home/ishan/code/lucene-solr-svn/lucene/core/src/java/org/apache/lucene/util/LegacyNumericUtils.java:244: error: invalid use of @throws
[javac] * @throws NumberFormatException if the supplied {@link BytesRef} is
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 55 errors
Any ideas, please? Apologies, I don't have much understanding of these doclint failures. Is there something wrong with my setup?
Hi Ishan Chattopadhyaya, I'm not sure why you are hitting that ... but can you email the dev list instead (dev@lucene.apache.org)?
Please do a complete "ant clean" from the root level of Lucene/Solr. Those errors generally happen (especially on Solr's build), if there are outdated class files still in around output folder or if a previous compile run failed.
Jenkins already checked that everything works with Java 8 and Java 9, so there should be no problem.
Did this ever hit the dev list? Google can't seem to find it. I'm wondering if a fix was ever found. Not because I'm seeing it when building Lucene - but because I'm seeing the same error when trying to build our own stuff. And clean does seem to stop it sometimes, but for some people it doesn't, so I'm trying to figure out whether anyone knows the actual cause. It's a bit of a vague error message.
This is mostly a "rote" rename, except there was some hair involved because I moved precisionStep and NumericType out of core's FieldType and down into the XXXField themselves.
This also required adding some deps on modules to backward-codecs, though I was able to cutover at least lucene/demo's usage to dimensional values ...