The javadoc for NetworkServer control says:
tracedirectory <tracedirectory> [-h <host>] [-p <portnumber>]: This changes where new trace files will be placed. For sessions with tracing already turned on, trace files remain in the previous location. Default is clousdcape.system.home
Apart from the problem of cloudscape.system.home instead of derby.system.home, when I turn tracing on the trace files are created in the current directory (user.dir) and not derby.system.home. Also if the documentation is correct, then the location when derby.system.home is not set should be specified.
java -Dderby.system.home=$PWD/dsh org.apache.derby.drda.NetworkServerControl start
Then turn tracing on with
java org.apache.derby.drda.NetworkServerControl trace on