I first noticed the problem using Java mail version 1.3.1 attempting to retrieve a malformed pop3 message from James. The message headers specify an invalid encoding type "Content-Transfer-Encoding: plain". When the command "TOP <msg> 0" is issued, James returns "+OK Message follows", then successfully returns all headers. Upon attempting to retrieve 0 lines of the message, it apparently encounters an error with that invalid encoding type and returns "-ERR Error while retrieving message." rather then the termination character of carriage return, period, carriage return. Javamail chokes on this waiting for the period termination character as it thinks that the "-ERR" text is a valid part of the message.
The "RETR" command works properly however and returns the message without error. Looking into the source code for James, I found that even though the argument is to return 0 lines of the message, James still attempts to write the message to the output with the call "mc.writeContentTo(nouts, lines);" lines = 0 in this case and I put in a temporary work around to not do this unless lines was > 0, but if a TOP command was issued with 5 for example, it would still return the invalid response.
Not sure what the best solution is, but it needs to either return -ERR up front, or to return the termination character regardless so that dumb mail clients can process it.