Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5217

DistCp fails when launched by Oozie in a secure cluster



    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.1.2
    • 1.3.0
    • distcp, security
    • None
    • Hadoop secure cluster

    • Reviewed


      As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
      in the main launcher for Pig, Hive, MR and Sqoop actions.

      if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
      jobConf.set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION"));

      For Java action, which does not have a main launcher in oozie, the above codecan be added by the user as the user purportedly has the code that is launched.

      But for DistCp action, the user has no such luxury. The solution attempted in
      MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it would break MAPREDUCE-3727. So, we have to fix DistCp and
      add the same boilerplate code so that DistCp action can be launched by Oozie
      in a secure cluster.

      The code added checks for an System env. variable to be set which is not
      typically set in normal command line execution of DistCp, DistCp runs fine
      with commnad line usage both in secure and non-secure cluster.


        1. MAPREDUCE-5217-trunk.patch
          1.0 kB
          Venkat Ranganathan
        2. MAPREDUCE-5217-trunk.patch
          1 kB
          Venkat Ranganathan
        3. MAPREDUCE-5217-branch-1.patch
          0.9 kB
          Venkat Ranganathan
        4. MAPREDUCE-5217-trunk.patch.2
          0.9 kB
          Venkat Ranganathan
        5. MAPREDUCE-5217-branch-1.patch.2
          0.8 kB
          Venkat Ranganathan
        6. MAPREDUCE-5217-trunk.patch.3
          1 kB
          Venkat Ranganathan
        7. MAPREDUCE-5217-branch-1.patch.3
          0.7 kB
          Venkat Ranganathan



            venkatnrangan Venkat Ranganathan
            venkatnrangan Venkat Ranganathan
            0 Vote for this issue
            5 Start watching this issue