Kafka release v0.9.0 and we should bump our Kafka support to be with that version.
This is related to
SPARK-12177 which pertains to adding support for the new kafka consumer API. However, this is a much, much smaller subset of that change, which simply bumps up the version of Kafka 0.9. We also use some non-public APIs of Kafka (AdminUtils, in particular) that have changed between the 2 releases, so this JIRA will take care of related fixes there as well.