[OPENNLP-1132] Fail with exception if not enough lines in leipzig parser

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.8.2
• Fix Version/s: 1.8.4
• Component/s: Language Detector
• Assignee: Peter Thygesen
• Reporter: Peter Thygesen
• Created: 14/Sep/17 09:07
• Updated: 26/Dec/17 18:47
• Resolved: 26/Dec/17 18:47

Description

Exception in thread "main" java.lang.IndexOutOfBoundsException: toIndex = 100000
at java.util.ArrayList.subListRangeCheck(ArrayList.java:1004)
at java.util.ArrayList.subList(ArrayList.java:996)
at opennlp.tools.formats.leipzig.LeipzigLanguageSampleStream$LeipzigSentencesStream.<init>(LeipzigLanguageSampleStream.java:65)
at opennlp.tools.formats.leipzig.LeipzigLanguageSampleStream.read(LeipzigLanguageSampleStream.java:157)
at opennlp.tools.formats.leipzig.LeipzigLanguageSampleStream.read(LeipzigLanguageSampleStream.java:42)
at opennlp.tools.formats.leipzig.SampleShuffleStream.<init>(SampleShuffleStream.java:38)
at opennlp.tools.formats.leipzig.LeipzigLanguageSampleStreamFactory.create(LeipzigLanguageSampleStreamFactory.java:76)
at opennlp.tools.cmdline.AbstractConverterTool.run(AbstractConverterTool.java:106)
at opennlp.tools.cmdline.CLI.main(CLI.java:256)

line 65:
Set<Integer> selectedLines = new HashSet<>(
indexes.subList(0, sentencesPerSample * numberOfSamples));

Fails if sentencesPerSample x numberOfSamples is larger than size of indexes (source file).

Issue Links

• links to: GitHub Pull Request #265