[TEZ-3240] Improvements to tez.lib.uris to allow for multiple tarballs and mixing tarballs and jars.

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.2, 0.9.0, 0.8.4
• Hadoop Flags: Reviewed
• Assignee: Eric Badger
• Reporter: Eric Badger
• Created: 04/May/16 15:04
• Updated: 15/Dec/16 16:01
• Resolved: 18/May/16 22:26

Description

Currently, tez.lib.uris only supports either a single archive or paths for multiple jars. You cannot mix and match between the two and you also cannot specify more than one archive. This means that you cannot specify both the tez and mapreduce archives. In the case where there is already a mapreduce archive in the distributed cache, you would not be able to use it when running tez. Instead, you would have to include the mapreduce jars in the single archive that you give to tez.lib.uris or use the mapreduce jars that are on the cluster node itself. This makes it very easy for the mapreduce versions to be out of sync with each other.

With the current implementation, during a rolling upgrade it is very easy to have jobs that do not get the same mapreduce jars across all of the containers, since some will start after the node's jars have been upgraded and some will start before.

If, instead, the job uses a archive that packages both tez and mapreduce together, then you will have 2 copies of the mapreduce jars in the distributed cache and will also have to upgrade both whenever you make a single upgrade to mapreduce.

I propose 2 improvements:
1) Allow tez.lib.uris to take an arbitrary number of archives and jars, while not being limited to only one or the other
2) Allow tez.lib.uris to specify a fragment following the '#' symbol (as is done in mapreduce) that will create a symlink with the name of the fragment.

Issue Links

• breaks: TEZ-3559 TEZ_LIB_URIS doesn't work with schemes different than the defaultFS