[OPENNLP-1131] LeipzigLanguageSampleStreamFactory should not load hidden files

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.8.2
• Fix Version/s: 1.8.3
• Component/s: Language Detector
• Assignee: Peter Thygesen
• Reporter: Peter Thygesen
• Created: 14/Sep/17 09:01
• Updated: 10/Oct/17 09:58
• Resolved: 10/Oct/17 09:58

Description

.DS_Store file is loaded as a sentence sample file. This is should not happen.

Exception in thread "main" java.io.UncheckedIOException: java.nio.charset.MalformedInputException: Input length = 1
at java.io.BufferedReader$1.hasNext(BufferedReader.java:574)
at java.util.Iterator.forEachRemaining(Iterator.java:115)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.LongPipeline.reduce(LongPipeline.java:438)
at java.util.stream.LongPipeline.sum(LongPipeline.java:396)
at java.util.stream.ReferencePipeline.count(ReferencePipeline.java:526)
at opennlp.tools.formats.leipzig.LeipzigLanguageSampleStream$LeipzigSentencesStream.<init>(LeipzigLanguageSampleStream.java:57)
at opennlp.tools.formats.leipzig.LeipzigLanguageSampleStream.read(LeipzigLanguageSampleStream.java:157)
at opennlp.tools.formats.leipzig.LeipzigLanguageSampleStream.read(LeipzigLanguageSampleStream.java:42)
at opennlp.tools.formats.leipzig.SampleShuffleStream.<init>(SampleShuffleStream.java:38)
at opennlp.tools.formats.leipzig.LeipzigLanguageSampleStreamFactory.create(LeipzigLanguageSampleStreamFactory.java:76)
at opennlp.tools.cmdline.AbstractConverterTool.run(AbstractConverterTool.java:106)
at opennlp.tools.cmdline.CLI.main(CLI.java:256)
Caused by: java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at java.io.BufferedReader$1.hasNext(BufferedReader.java:571)
... 16 more

Issue Links

• links to: GitHub Pull Request #264