Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3727

jobtoken location property in jobconf refers to wrong jobtoken file

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 1.0.0
    • Fix Version/s: 0.23.1, 1.2.0
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the HADOOP_TOKEN_FILE_LOCATION ENV var and seeds it as the mapreduce.job.credentials.binary property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.

      The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected mapreduce.job.credentials.binary property to load the credentials and submit their MR jobs.

      The problem is that the mapreduce.job.credentials.binary property also makes it to the tasks of the MR/Pig/Hive/Sqoop MR jobs.

      If for some reason the MR/Pig/Hive/Sqoop MR code does some logic that triggers the credential loading, because the property is set, the credential loading fails trying to load a jobtoken file of the launcher job which does not exists in the context of the MR/Pig/Hive/Sqoop jobs.

      More specifically, we are seeing this happening with certain hive queries that trigger a conditional code within their RowContainer which then uses the FileInputFormat.getSplits() and then the TokenCache tries to load credentials for a file that is for the wrong job.

      1. MAPREDUCE-3727-branch1.patch
        3 kB
        Alejandro Abdelnur
      2. MAPREDUCE-3727.patch
        4 kB
        Alejandro Abdelnur

        Issue Links

          Activity

          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Harsh J made changes -
          Link This issue is broken by MAPREDUCE-4324 [ MAPREDUCE-4324 ]
          Alejandro Abdelnur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Target Version/s 1.1.1 [ 12321660 ]
          Fix Version/s 1.2.0 [ 12321661 ]
          Resolution Fixed [ 1 ]
          Harsh J made changes -
          Fix Version/s 0.23.1 [ 12318883 ]
          Fix Version/s 0.23.0 [ 12315570 ]
          Harsh J made changes -
          Affects Version/s 0.23.0 [ 12315570 ]
          Affects Version/s 0.23.1 [ 12318883 ]
          Harsh J made changes -
          Fix Version/s 0.23.0 [ 12315570 ]
          Fix Version/s 0.24.0 [ 12317654 ]
          Fix Version/s 1.1.0 [ 12317960 ]
          Target Version/s 1.1.1 [ 12321660 ]
          Arun C Murthy made changes -
          Fix Version/s 0.24.0 [ 12317654 ]
          Fix Version/s 0.23.1 [ 12318883 ]
          Alejandro Abdelnur made changes -
          Attachment MAPREDUCE-3727-branch1.patch [ 12513213 ]
          Alejandro Abdelnur made changes -
          Link This issue is blocked by HADOOP-8023 [ HADOOP-8023 ]
          Alejandro Abdelnur made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Alejandro Abdelnur made changes -
          Attachment MAPREDUCE-3727.patch [ 12513027 ]
          Alejandro Abdelnur made changes -
          Fix Version/s 0.23.1 [ 12318883 ]
          Affects Version/s 0.23.1 [ 12318883 ]
          Alejandro Abdelnur made changes -
          Field Original Value New Value
          Fix Version/s 1.1.0 [ 12317960 ]
          Alejandro Abdelnur created issue -

            People

            • Assignee:
              Alejandro Abdelnur
              Reporter:
              Alejandro Abdelnur
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development