Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-1619 Enhancing sharelib support
  3. OOZIE-3227

Eliminate duplicate dependencies when using Hadoop 3 DistributedCache

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 5.0.0, 4.3.1
    • Fix Version/s: 5.1.0
    • Component/s: core
    • Labels:
      None

      Description

      Using Hadoop 3 it is not allowed to have multiple dependencies with same file names on the list of mapreduce.job.cache.files.

      The issue occurs when I have the same file name on multiple sharelib folders and/or my application's lib folder. This can be avoided but not easy all the time.

      I suggest to remove the duplicates from this list.
      A quick workaround for the source code in JavaActionExecutor is like:

                  removeDuplicatedDependencies(launcherJobConf, "mapreduce.job.cache.files");
                  removeDuplicatedDependencies(launcherJobConf, "mapreduce.job.cache.archives");
      ......
      private void removeDuplicatedDependencies(JobConf conf, String key) {
              final Map<String, String> nameToPath = new HashMap<>();
              StringBuilder uniqList = new StringBuilder();
              for(String dependency: conf.get(key).split(",")) {
                  final String[] arr = dependency.split("/");
                  final String dependencyName = arr[arr.length - 1];
                  if(nameToPath.containsKey(dependencyName)) {
                      LOG.warn(dependencyName + " [" + dependency + "] is already defined in " + key + ". Skipping...");
                  } else {
                      nameToPath.put(dependencyName, dependency);
                      uniqList.append(dependency).append(",");
                  }
              }
              uniqList.setLength(uniqList.length() - 1);
              conf.set(key, uniqList.toString());
          }
      

      Other way is to eliminate the deprecated org.apache.hadoop.filecache.DistributedCache.

      I am going to have a deeper understanding how we should use distributed cache and all the comments are welcome.

        Attachments

        1. OOZIE-3227_001.patch
          14 kB
          Dénes Bodó
        2. OOZIE-3227_002.patch
          17 kB
          Dénes Bodó

          Issue Links

            Activity

              People

              • Assignee:
                dionusos Dénes Bodó
                Reporter:
                dionusos Dénes Bodó
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: