[OPENNLP-1357] Use CharSequence to allow for memory management

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.9.4
• Fix Version/s: 2.1.1
• Component/s: Sentence Detector
• Assignee: Martin Wiesner
• Reporter: Paul Austin
• Created: 14/Mar/22 18:26
• Updated: 28/Jan/23 14:59
• Resolved: 15/Dec/22 07:10

Description

Most of the classes in OpenNLP require the inputs to be as String, StringBuffer, or char[]. This means that you have to load all the data into memory.

Many of these cases (String and StringBuffer args) could be replaced with a single method that accepts CharSequence as a parameter.

For example DefaultEndOfSentenceScanner

 

 public List<Integer> getPositions(CharSequence s) {
    List<Integer> l = new ArrayList<>();
    for (int i = 0; i < s.length(); i++) {
      char c = s.charAt(i);
      if (eosCharacters.contains(c)) {
        l.add(i);
      }
    }
    return l;
  }

This would allow for users to manage the memory overhead for large data sets. And in some cases require less temporary memory conversion to char buffers.

Some code such as the SDContextGenerator already uses CharSequence.  However in SentenceDetectorME there is an unnecessary conversion to a StringBuffer. The sb isn't modified and the SDContextGenerator.getContext takes CharSequence as an arg and String is a CharSequence.

 

public Span[] sentPosDetect(String s) {
    sentProbs.clear();
    StringBuffer sb = new StringBuffer(s);

 

I can create a pull request(s) for the above if you think it is useful.