Calling awaitTermination shouldn't be necessary, because invokeAll() won't return until everything has finished, but it won't hurt anything and is a good check to have. I'll add it in.
I didn't use stopJettySolrRunner() because it relies on the position of a Jetty in the jettys list, which will not be constant if we're calling it in parallel. It's probably worth clearing the list after everything's done though.
Error handling is a slightly tricky one. Should we fail startup if one of the jetties doesn't start? The behaviour at the moment isn't great - you'll get an exception if one of the starting jetties dies, but the already-started-up ones will keep running. At close, I guess the nicest behaviour would be to collect all exceptions from the futures, and throw an IOException with any collected exceptions added as suppressed.