[LUCENE-9215] replace checkJavaDocs.py with doclet

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 9.0
• Lucene Fields: New
• Assignee: Robert Muir
• Reporter: Robert Muir
• Created: 08/Feb/20 15:27
• Updated: 28/Aug/22 15:58
• Resolved: 02/Sep/20 12:38

Description

The current checker runs regular expressions against html, and it breaks when newer java change html output. This is not particularly fun to fix: see LUCENE-9213

Java releases often now, and when i compared generated html of a simple class across 11,12,13 it surprised me how much changes. So I think we want to avoid parsing their HTML.

Javadoc Xdoclint feature has a "missing checker": but it is black/white. Either everything is fully documented or its not. And while you can enable/disable doclint checks per-package, this also seems black/white (either all checks or no checks at all).

On the other hand the python checker is able to check per-package at different granularities (package, class, method). It makes it possible to iteratively improve the situation.

With doclet api we could implement checks in pure java, for example to match checkJavaDocs.py logic:

  private void checkComment(Element element) {
    var tree = docTrees.getDocCommentTree(element);
    if (tree == null) {
      error(element, "javadocs are missing");
    } else {
      var normalized = tree.getFirstSentence().get(0).toString()
                       .replace('\u00A0', ' ')
                       .trim()
                       .toLowerCase(Locale.ROOT);
      if (normalized.isEmpty()) {
        error(element, "blank javadoc comment");
      } else if (normalized.startsWith("licensed to the apache software foundation") ||
                 normalized.startsWith("copyright 2004 the apache software foundation")) {
        error(element, "comment is really a license");
      }
    }

If there are problems then they just appear as errors from the output of javadoc like usual:

javadoc: error - org.apache.lucene.nodoc (package): javadocs are missing
/home/rmuir/workspace/lucene-solr/lucene/core/src/java/org/apache/lucene/search/spans/SpanNearQuery.java:190: error - SpanNearWeight (class): javadocs are missing
/home/rmuir/workspace/lucene-solr/lucene/core/src/java/org/apache/lucene/search/spans/SpanContainingQuery.java:54: error - SpanContainingWeight (class): javadocs are missing
/home/rmuir/workspace/lucene-solr/lucene/core/src/java/org/apache/lucene/search/spans/SpanWithinQuery.java:55: error - SpanWithinWeight (class): javadocs are missing
/home/rmuir/workspace/lucene-solr/lucene/core/src/java/org/apache/lucene/search/spans/SpanTermQuery.java:94: error - SpanTermWeight (class): javadocs are missing
/home/rmuir/workspace/lucene-solr/lucene/core/src/java/org/apache/lucene/search/spans/SpanNotQuery.java:109: error - SpanNotWeight (class): javadocs are missing
/home/rmuir/workspace/lucene-solr/lucene/core/src/java/org/apache/lucene/search/spans/SpanOrQuery.java:139: error - SpanOrWeight (class): javadocs are missing
/home/rmuir/workspace/lucene-solr/lucene/core/src/java/org/apache/lucene/search/spans/SpanPositionCheckQuery.java:77: error - SpanPositionCheckWeight (class): javadocs are missing
/home/rmuir/workspace/lucene-solr/lucene/core/src/java/org/apache/lucene/search/MultiCollectorManager.java:61: error - Collectors (class): javadocs are missing
/home/rmuir/workspace/lucene-solr/lucene/core/src/java/org/apache/lucene/search/MultiCollectorManager.java:89: error - LeafCollectors (class): javadocs are missing
/home/rmuir/workspace/lucene-solr/lucene/core/src/java/org/apache/lucene/util/PagedBytes.java:353: error - PagedBytesDataOutput (class): javadocs are missing
/home/rmuir/workspace/lucene-solr/lucene/core/src/java/org/apache/lucene/util/PagedBytes.java:285: error - PagedBytesDataInput (class): javadocs are missing
/home/rmuir/workspace/lucene-solr/lucene/core/src/java/org/apache/lucene/nodoc/EmptyDoc.java:22: error - EmptyDoc (class): javadocs are missing
/home/rmuir/workspace/lucene-solr/lucene/core/src/java/org/apache/lucene/nodoc/LicenseDoc.java:36: error - LicenseDoc (class): comment is really a license
/home/rmuir/workspace/lucene-solr/lucene/core/src/java/org/apache/lucene/nodoc/NoDoc.java:19: error - NoDoc (class): javadocs are missing

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':lucene:core:javadoc'.
> Javadoc generation failed. Generated Javadoc options file (useful for troubleshooting): '/home/rmuir/workspace/lucene-solr/lucene/core/build/tmp/javadoc/javadoc.options'

Issue Links

• is related to: LUCENE-9496 Replace (or accelerate) check-broken-links.gradle with a doclet pass
• links to: GitHub Pull Request #1802