With JDK9 the behaviour of javadoc has apparently changed (at least on Windows platform).
In a project with special characters (umlaut) in the organization name the content of the options file was written with the default platform encoding (cp1252).
As a result, the javadoc generation fails on javadoc execution with "Exit code: 1 - javadoc: error - cannot read Input length = 1".
With JDK8 javadoc generation works.
It also succeeds with JDK9 and setting the environment variable: JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF-8
However, this should not be necessary, and maven-javadoc-plugin should work out of the box.
After some tests it appears to be the case that the JDK9 Javadoc reads @-files always with UTF-8 charset, regardless of platform encoding.
I looked into the issue and prepared a patch for the plugin when working with JDK9+ and added two tests for the temporary files options and argfile.
To my knowledge, the temporary file "packages" cannot contain special characters, since this isn't allowed in the java specification, so no workaround is added.
The file javadoc.bat is not affected (a test with a JDK in the directory "C:\jdk9äöüß" worked with the file being platform encoded).
I'll try to create a pull request from https://github.com/michael-st/maven-javadoc-plugin