Apache Kafka release 0.9.0.0 came out some time and we should add support for it. This JIRA is related to
SPARK-12177 which is related to adding support for new consumer API only available starting v0.9.0.0
However, we should upgrade Kafka to 0.9.0.0 regardless of when (and before) the support for the new consumer API gets added.
We also use some non-public APIs from Kafka which have changed in 0.9.0.0 release. So, this change should also take care of updating those usages.