Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5217

DistCp fails when launched by Oozie in a secure cluster

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.1.2
    • 1.3.0
    • distcp, security
    • None
    • Hadoop secure cluster

    • Reviewed

    Description

      As mentioned in MAPREDUCE-4324, Oozie has the following boilerplate code in
      in the main launcher for Pig, Hive, MR and Sqoop actions.

      if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
      jobConf.set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
      }

      For Java action, which does not have a main launcher in oozie, the above codecan be added by the user as the user purportedly has the code that is launched.

      But for DistCp action, the user has no such luxury. The solution attempted in
      MAPREDUCE-4324 would have helped DistCp, but it was not implemented as it would break MAPREDUCE-3727. So, we have to fix DistCp and
      add the same boilerplate code so that DistCp action can be launched by Oozie
      in a secure cluster.

      The code added checks for an System env. variable to be set which is not
      typically set in normal command line execution of DistCp, DistCp runs fine
      with commnad line usage both in secure and non-secure cluster.

      Attachments

        1. MAPREDUCE-5217-branch-1.patch.3
          0.7 kB
          Venkat Ranganathan
        2. MAPREDUCE-5217-trunk.patch.3
          1 kB
          Venkat Ranganathan
        3. MAPREDUCE-5217-branch-1.patch.2
          0.8 kB
          Venkat Ranganathan
        4. MAPREDUCE-5217-trunk.patch.2
          0.9 kB
          Venkat Ranganathan
        5. MAPREDUCE-5217-branch-1.patch
          0.9 kB
          Venkat Ranganathan
        6. MAPREDUCE-5217-trunk.patch
          1 kB
          Venkat Ranganathan
        7. MAPREDUCE-5217-trunk.patch
          1.0 kB
          Venkat Ranganathan

        Activity

          People

            venkatnrangan Venkat Ranganathan
            venkatnrangan Venkat Ranganathan
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: