Thanks Mani! This is looking great! I have a few more comments, my apologies, I missed them in the first run.
1. do-component-build checks for presence of GIT_REPO and if it's set, creates a temporary tarball. I don't think that's how the current Bigtop build workflow works. We don't ever define GIT_REPO, so I'd suggest getting rid of the bottom else clause completely. Would you agree?
2. I noticed that zookeeper-server is now a dependency of kafka-server and zookeeper is dependency of kafka package. I think this is much better than the previous patch. And, while I totally agree with the dependency of kafka on zookeeper I don't feel that the kafka-server needs to depend on zookeeper-server. The reason is that kafka just needs to find a zookeeper ensemble, for that you don't need to have the ensemble running on the same nodes as kafka. So, simply having a dependency of kafka on zookeeper (which enables it to connect to a zookeeper ensemble - whether on the same node or a different node), should be enough. If people want to run a zookeeper server on the same node, they will follow the zookeeper documentation to install, setup and run a zookeeper server on that node and end up install zookeeper-server package on it anyways. We don't have to install it automatically.
3. I noticed that when we create the kafka user, we do:
getent passwd kafka > /dev/null || useradd -c "kafka" -s /bin/bash -g kafka -d /var/lib/kafka kafka 2> /dev/null || :
The long name here is 'kafka'. It's not a big deal, but historically, our convention has been to have long name starting with an upper case letter so that would make it 'Kafka'. See spark, hadoop and hive, for example.
4. In the .mk file, I noticed:
Is that intentional?
And, this is a question than a suggestion:
In line 125 of install_kafka.sh, we have:
for file in kafka-console-consumer.sh kafka-console-producer.sh kafka-run-class.sh kafka-topics.sh
That looks fragile in case new binaries get added to kafka's bin directory. Are there any .sh files in that directory that we don't want to create shell wrappers for? Is that why we are spelling them out?