"It isn't normal for the JVM to take memory and not give it back."
Yes it is, the JVM will take up to -Xmx and not release it ever.
"There is a garbage collector, and it should manage the size of the heap to a reasonable size."
The garbage collection manages the heap contents and can trigger fresh allocations of OS memory for the heap to grow into, but cannot reduce the memory allocated to the heap by the JVM/OS
"That it isn't doing so could mean that resources are not being released properly,"
Not correct, if the amount of heap space in-use increased and never decreased this would be true, but you are talking about the size of the availbe memeory, not the heap contents.
" or that the GC doesn't get into action, and it can be forced into action, "
It cannot be forced, only suggested, and it is considered bad practice for an application to interfere with garbage collection.
"and it should act if James is sitting for a day on 300MB and a few seconds of operation time."
Garbage collection is only carried out when an allocation is requested from the heap which would cause one or more of the heap partitions (young, tenured, permanent) to exceed a threshold % of their current size. If this collection doesn't result in enough space free in the heap it will trigger an increase in partition or heap size up to the absolute maximum size of the partition or the total heap, whichever is reached first.
James is multi threaded, and so will require heap space for all active processes.
Mailets are allowed, expected even, to examine and modify the content of messages. Therfore it is not unreasonable to use JavaMail as the message model. The draw back is that MimeMessage will read and decode messages "eagerly", largely because JavaMail is primarily a client API and Sun have expressed the opinion that they would not consider altering JavaMail to better suit the requirements of server applications.
The solution is for someone to create new JavaMail content handlers. this is not a trivial task, but is one that the James commiters have considered in the past.