[NIFI-2787] PersistentProvenanceRepository rollover can fail on immense indexed attributes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0, 0.7.0
• Fix Version/s: 1.1.0, 0.7.1
• Component/s: Core Framework
• Assignee: Michael W Moser
• Reporter: Michael W Moser
• Created: 16/Sep/16 21:51
• Updated: 27/Sep/16 20:17
• Resolved: 27/Sep/16 20:17

Description

Accidentally created an immense attribute (36,000 bytes), which I indexed with nifi.provenance.repository.indexed.attributes. Received this error.

ERROR [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Failed to rollover Provenance repository due to java.lang.IllegalArgumentException: Document contains at least one immense term in field="FOO" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms.

Perhaps this is as simple as changing https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/RepositoryConfiguration.java#L37 to 32766 to match Lucene. Investigation & testing needed.

For background, this Lucene ticket made exceeding the term size limit an IllegalArgumentException https://issues.apache.org/jira/browse/LUCENE-5472

Issue Links

• links to: GitHub Pull Request #1043