Major Description
Packages should depend on packaged .jar files, instead of including copies of the jars in every package again and again.
Most packages would shrink substantially in size once these redundancies have been removed. From a quick test, about 18 MB of a 20 MB package are shareable .jar files!
Apache Commons, Jackson, Protobuf, etc.
all these exist as operating system packages (with security support etc.) on most Linux distributions. There is no need to include copies of them inside of hadoop packages (in fact, this violates the packaging policy of e.g. Ubuntu and Debian)
Even worse, the files are sometimes even included more than once (and they definitely are included in more than one package).
Where possible, the lib/ folder should consist of symlinks to the appropriate jar files in /usr/share/java, and dependencies should be used to ensure these jar files exist on the system. This way, they can be shared across different applications.