Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3499

New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.1, 0.24.0
    • Fix Version/s: 0.23.1
    • Component/s: mrv2, test
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      The new MiniMR implementation is not taking proxyuser settings.

      Because of this, testcases using/testing doAs functionality fail.

      This affects all Oozie testcases that use MiniMR.

      1. TestForMAPREDUCE-3499.patch
        6 kB
        Alejandro Abdelnur
      2. MR-3499.patch
        10 kB
        John George
      3. MR-3499.patch
        8 kB
        Mahadev konar
      4. MR-3499.patch
        14 kB
        Mahadev konar
      5. MR-3499.patch
        14 kB
        John George
      6. MR-3499.patch
        7 kB
        John George
      7. MR-3499.patch
        14 kB
        John George

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #977 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/977/)
        MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu)

        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239207
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #977 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/977/ ) MAPREDUCE-3499 . New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239207 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #179 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/179/)
        Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499

        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239210
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #179 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/179/ ) Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239210 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #157 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/157/)
        Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499

        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239210
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #157 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/157/ ) Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239210 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #944 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/944/)
        MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu)

        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239207
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #944 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/944/ ) MAPREDUCE-3499 . New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239207 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #474 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/474/)
        Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499

        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239210
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #474 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/474/ ) Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239210 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1648 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1648/)
        MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu)

        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239207
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1648 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1648/ ) MAPREDUCE-3499 . New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239207 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1632 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1632/)
        MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu)

        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239207
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1632 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1632/ ) MAPREDUCE-3499 . New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239207 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #450 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/450/)
        Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499

        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239210
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #450 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/450/ ) Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239210 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #460 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/460/)
        Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499

        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239210
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #460 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/460/ ) Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239210 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1703 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1703/)
        MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu)

        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239207
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1703 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1703/ ) MAPREDUCE-3499 . New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1239207 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java
        Hide
        Alejandro Abdelnur added a comment -

        Thanks John & Mahadev for the work on this one. Committed to trunk and branch-0.23

        Show
        Alejandro Abdelnur added a comment - Thanks John & Mahadev for the work on this one. Committed to trunk and branch-0.23
        Hide
        Alejandro Abdelnur added a comment -

        +1

        Show
        Alejandro Abdelnur added a comment - +1
        Hide
        John George added a comment -

        Sorry for that. Attaching a patch with all files included.

        Show
        John George added a comment - Sorry for that. Attaching a patch with all files included.
        Hide
        Alejandro Abdelnur added a comment -

        John, the patch is missing the testcase.

        Show
        Alejandro Abdelnur added a comment - John, the patch is missing the testcase.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12512712/MR-3499.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1731//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1731//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512712/MR-3499.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1731//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1731//console This message is automatically generated.
        Hide
        John George added a comment -

        Thanks Alejandro. Removed trailing spaces from Test program. I could not see any indentation issues in YARNRunner.

        Show
        John George added a comment - Thanks Alejandro. Removed trailing spaces from Test program. I could not see any indentation issues in YARNRunner.
        Hide
        Alejandro Abdelnur added a comment -

        Mahadev, John,

        Apologies for the delay. I could not reproduce the problem (the Counters$Group related exception) until Tom gave me a hint and thus the solution. I'll update MAPREDUCE-3697 with my the details of the Counters$Group exception.

        The problem I'm seeing now is that I'm getting different Hadoop built in counters, before I used to get a "Task$Counter" counter, now I'm getting "org.apache.hadoop.mapreduce.TaskCounter"

        But other than that, things seem to work for what this patch is fixing.

        Thanks.

        Patch looks good. A couple of minor code-style things, YARNRunner changes have wrong indentation and the TestMiniMRProxyUser has some trailing spaces.

        Show
        Alejandro Abdelnur added a comment - Mahadev, John, Apologies for the delay. I could not reproduce the problem (the Counters$Group related exception) until Tom gave me a hint and thus the solution. I'll update MAPREDUCE-3697 with my the details of the Counters$Group exception. The problem I'm seeing now is that I'm getting different Hadoop built in counters, before I used to get a "Task$Counter" counter, now I'm getting "org.apache.hadoop.mapreduce.TaskCounter" But other than that, things seem to work for what this patch is fixing. Thanks. Patch looks good. A couple of minor code-style things, YARNRunner changes have wrong indentation and the TestMiniMRProxyUser has some trailing spaces.
        Hide
        Mahadev konar added a comment -

        @Alejandro,
        Any pointers to re create the counters issue? Trying to fix it but would help if I can test the fix easily via oozie.

        Show
        Mahadev konar added a comment - @Alejandro, Any pointers to re create the counters issue? Trying to fix it but would help if I can test the fix easily via oozie.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12512598/MR-3499.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 3721 javac compiler warnings (more than the trunk's current 3706 warnings).

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1725//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1725//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512598/MR-3499.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 3721 javac compiler warnings (more than the trunk's current 3706 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1725//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1725//console This message is automatically generated.
        Hide
        Mahadev konar added a comment -

        Thanks Alejandro. I am working on a patch for MAPREDUCE-3697. Will update soon. Can you please verify John's latest patch?

        Show
        Mahadev konar added a comment - Thanks Alejandro. I am working on a patch for MAPREDUCE-3697 . Will update soon. Can you please verify John's latest patch?
        Hide
        John George added a comment -

        Thanks for the update Alejandro. I will upload a patch with changes suggested by you, Sid and include Mahadev's patch in the next few minutes.

        Show
        John George added a comment - Thanks for the update Alejandro. I will upload a patch with changes suggested by you, Sid and include Mahadev's patch in the next few minutes.
        Hide
        Alejandro Abdelnur added a comment -

        Mahadev,

        I was missing something, now I got things working. I think we are good with this patch for now.

        I'm able to successfully submit job to MiniCluster from Oozie, I'm now hitting the Counter's API change, MAPREDUCE-3697

        java.lang.NoSuchMethodError: org.apache.hadoop.mapred.Counters.getGroup(Ljava/lang/String;)Lorg/apache/hadoop/mapred/Counters$Group;
        	at org.apache.oozie.action.hadoop.LauncherMapper.hasIdSwap(LauncherMapper.java:273)
        	at org.apache.oozie.action.hadoop.TestMapReduceActionExecutor.testSetExecutionStats_when_user_has_specified_stats_write_FALSE(TestMapReduceActionExecutor.java:562)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at com.intellij.junit3.JUnit3IdeaTestRunner.doRun(JUnit3IdeaTestRunner.java:139)
        	at com.intellij.junit3.JUnit3IdeaTestRunner.startRunnerWithArgs(JUnit3IdeaTestRunner.java:52)
        	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:199)
        	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:62)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
        

        Thanks.

        Alejandro

        Show
        Alejandro Abdelnur added a comment - Mahadev, I was missing something, now I got things working. I think we are good with this patch for now. I'm able to successfully submit job to MiniCluster from Oozie, I'm now hitting the Counter's API change, MAPREDUCE-3697 java.lang.NoSuchMethodError: org.apache.hadoop.mapred.Counters.getGroup(Ljava/lang/ String ;)Lorg/apache/hadoop/mapred/Counters$Group; at org.apache.oozie.action.hadoop.LauncherMapper.hasIdSwap(LauncherMapper.java:273) at org.apache.oozie.action.hadoop.TestMapReduceActionExecutor.testSetExecutionStats_when_user_has_specified_stats_write_FALSE(TestMapReduceActionExecutor.java:562) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at com.intellij.junit3.JUnit3IdeaTestRunner.doRun(JUnit3IdeaTestRunner.java:139) at com.intellij.junit3.JUnit3IdeaTestRunner.startRunnerWithArgs(JUnit3IdeaTestRunner.java:52) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:199) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:62) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120) Thanks. Alejandro
        Hide
        Mahadev konar added a comment -

        @Alejandro,
        Weird. This is coming from an unsupported file system. Is there someway I can try it out and debug on my own?

        Show
        Mahadev konar added a comment - @Alejandro, Weird. This is coming from an unsupported file system. Is there someway I can try it out and debug on my own?
        Hide
        Alejandro Abdelnur added a comment -

        I'm getting a bit further, now I'm getting the following messages

        22:04:34,129  WARN Configuration:639 - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
        22:04:34,133  WARN Configuration:639 - fs.default.name is deprecated. Instead, use fs.defaultFS
        22:04:34,135  WARN Configuration:639 - user.name is deprecated. Instead, use mapreduce.job.user.name
        22:04:34,139 ERROR UserGroupInformation:1180 - PriviledgedActionException as:test (auth:PROXY) via tucu (auth:SIMPLE) cause:org.apache.hadoop.fs.UnsupportedFileSystemException: No AbstractFileSystem for scheme: hdfs
        22:04:34,141  INFO Cluster:116 - Failed to use org.apache.hadoop.mapred.YarnClientProtocolProvider due to error: Error in instantiating YarnClient
        22:04:34,141 ERROR UserGroupInformation:1180 - PriviledgedActionException as:test (auth:PROXY) via tucu (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
        
        Show
        Alejandro Abdelnur added a comment - I'm getting a bit further, now I'm getting the following messages 22:04:34,129 WARN Configuration:639 - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 22:04:34,133 WARN Configuration:639 - fs. default .name is deprecated. Instead, use fs.defaultFS 22:04:34,135 WARN Configuration:639 - user.name is deprecated. Instead, use mapreduce.job.user.name 22:04:34,139 ERROR UserGroupInformation:1180 - PriviledgedActionException as:test (auth:PROXY) via tucu (auth:SIMPLE) cause:org.apache.hadoop.fs.UnsupportedFileSystemException: No AbstractFileSystem for scheme: hdfs 22:04:34,141 INFO Cluster:116 - Failed to use org.apache.hadoop.mapred.YarnClientProtocolProvider due to error: Error in instantiating YarnClient 22:04:34,141 ERROR UserGroupInformation:1180 - PriviledgedActionException as:test (auth:PROXY) via tucu (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
        Hide
        Mahadev konar added a comment -

        @Alejandro,
        Any more details on the issues?

        Show
        Mahadev konar added a comment - @Alejandro, Any more details on the issues?
        Hide
        Alejandro Abdelnur added a comment -

        Mahadev's tweaks get me a bit futher, but I'm still hitting some other issues. still digging.

        Thanks.

        Show
        Alejandro Abdelnur added a comment - Mahadev's tweaks get me a bit futher, but I'm still hitting some other issues. still digging. Thanks.
        Hide
        John George added a comment -

        @Mahadev - I will incorporate your patch and submit a new one with the changes alejandro proposed

        @Sid - How does this work in a real cluster? I mean when oozie submits a job as a proxy user, I dint think the user had to be the same group or user? Or am i wrong?

        Show
        John George added a comment - @Mahadev - I will incorporate your patch and submit a new one with the changes alejandro proposed @Sid - How does this work in a real cluster? I mean when oozie submits a job as a proxy user, I dint think the user had to be the same group or user? Or am i wrong?
        Hide
        Siddharth Seth added a comment -

        John, I took a very brief look at the patch. Changing the default history dir permissions is avoidable - they were specifically setup to prevent users having access to others history files. For MiniCluster, maybe these directories can pre-created with the required permissions?

        Show
        Siddharth Seth added a comment - John, I took a very brief look at the patch. Changing the default history dir permissions is avoidable - they were specifically setup to prevent users having access to others history files. For MiniCluster, maybe these directories can pre-created with the required permissions?
        Hide
        Mahadev konar added a comment -

        I had forgotten to attach the test file in the previous patch. This patch has the test case.

        I havent updated the patch that Alejandro suggested. John, you want to take it from here?

        Show
        Mahadev konar added a comment - I had forgotten to attach the test file in the previous patch. This patch has the test case. I havent updated the patch that Alejandro suggested. John, you want to take it from here?
        Hide
        Mahadev konar added a comment -

        Updated the patch with my changes on top of John's changes.

        @Alejandro, want to give this a try?

        Show
        Mahadev konar added a comment - Updated the patch with my changes on top of John's changes. @Alejandro, want to give this a try?
        Hide
        Alejandro Abdelnur added a comment -

        Hi Mahadev, any luck with this?

        Show
        Alejandro Abdelnur added a comment - Hi Mahadev, any luck with this?
        Hide
        Mahadev konar added a comment -

        I think I know what the issue above is. Ill upload an updated patch with the fix by tonight.

        Show
        Mahadev konar added a comment - I think I know what the issue above is. Ill upload an updated patch with the fix by tonight.
        Hide
        Alejandro Abdelnur added a comment -

        I've applied this patch locally and tested Oozie with it. The patch seems to do the trick to pass the proxyuser issue.

        However I'm not running into the following error when trying run an Oozie testcase that submits a MR job to MiniMR:

        testRecovery(org.apache.oozie.action.hadoop.TestJavaActionExecutor)  Time elapsed: 0.994 sec  <<< ERROR!
        org.apache.oozie.action.ActionExecutorException: NullPointerException: null
        	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:386)
        	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:670)
        	at org.apache.oozie.action.hadoop.TestJavaActionExecutor.submitAction(TestJavaActionExecutor.java:326)
        	at org.apache.oozie.action.hadoop.TestJavaActionExecutor.testRecovery(TestJavaActionExecutor.java:578)
        Caused by: java.lang.NullPointerException
        	at org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:193)
        	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:270)
        	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:379)
        	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
        	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1212)
        	at java.security.AccessController.doPrivileged(Native Method)
        	at javax.security.auth.Subject.doAs(Subject.java:396)
        	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
        	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
        	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:611)
        	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:606)
        	at java.security.AccessController.doPrivileged(Native Method)
        	at javax.security.auth.Subject.doAs(Subject.java:396)
        	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
        	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:606)
        	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:655)
        	... 30 more
        

        Is this still related or ...?

        Show
        Alejandro Abdelnur added a comment - I've applied this patch locally and tested Oozie with it. The patch seems to do the trick to pass the proxyuser issue. However I'm not running into the following error when trying run an Oozie testcase that submits a MR job to MiniMR: testRecovery(org.apache.oozie.action.hadoop.TestJavaActionExecutor) Time elapsed: 0.994 sec <<< ERROR! org.apache.oozie.action.ActionExecutorException: NullPointerException: null at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:386) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:670) at org.apache.oozie.action.hadoop.TestJavaActionExecutor.submitAction(TestJavaActionExecutor.java:326) at org.apache.oozie.action.hadoop.TestJavaActionExecutor.testRecovery(TestJavaActionExecutor.java:578) Caused by: java.lang.NullPointerException at org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:193) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:270) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:379) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1212) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:611) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:606) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:606) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:655) ... 30 more Is this still related or ...?
        Hide
        Alejandro Abdelnur added a comment -

        John,

        A couple of things:

        #1

        You changed the setting for the staging dir of MiniMRYarnCluster to '/tmp'. Why not the original resolution to the getTestWorkingDir() ? I know it resolves to 'target/...' and in projects like Pig/Hive which use ant and 'build/' dir this would break; but should you create a fresh dir under /tmp then? (using the File.createTempFile() method -you have to create the file, then delete it and then create a dir with that name).

        #2

        I had to tweak the test (how to create the list of valid hosts) to get it to work:

        Instead doing conf.set("hadoop.proxyuser." + proxyUser + ".hosts", "10.74.90.219,127.0.0.1"); I've done:

            StringBuilder sb = new StringBuilder();
            sb.append("127.0.0.1,localhost");
            for (InetAddress i : InetAddress.getAllByName(InetAddress.getLocalHost().getHostName())) {
              sb.append(",").append(i.getCanonicalHostName());
            }
            conf.set("hadoop.proxyuser." + proxyUser + ".hosts", sb.toString());
        

        And then the test works.

        Thanks.

        Show
        Alejandro Abdelnur added a comment - John, A couple of things: #1 You changed the setting for the staging dir of MiniMRYarnCluster to '/tmp'. Why not the original resolution to the getTestWorkingDir() ? I know it resolves to 'target/...' and in projects like Pig/Hive which use ant and 'build/' dir this would break; but should you create a fresh dir under /tmp then? (using the File.createTempFile() method -you have to create the file, then delete it and then create a dir with that name). #2 I had to tweak the test (how to create the list of valid hosts) to get it to work: Instead doing conf.set("hadoop.proxyuser." + proxyUser + ".hosts", "10.74.90.219,127.0.0.1"); I've done: StringBuilder sb = new StringBuilder(); sb.append( "127.0.0.1,localhost" ); for (InetAddress i : InetAddress.getAllByName(InetAddress.getLocalHost().getHostName())) { sb.append( "," ).append(i.getCanonicalHostName()); } conf.set( "hadoop.proxyuser." + proxyUser + ".hosts" , sb.toString()); And then the test works. Thanks.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12512020/MR-3499.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 3732 javac compiler warnings (more than the trunk's current 3717 warnings).

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1687//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1687//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512020/MR-3499.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 3732 javac compiler warnings (more than the trunk's current 3717 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1687//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1687//console This message is automatically generated.
        Hide
        John George added a comment -

        Uploading a patch that passes the attached test by Alejandro. Not sure if this is the best way, hence comments or suggestions welcome

        Show
        John George added a comment - Uploading a patch that passes the attached test by Alejandro. Not sure if this is the best way, hence comments or suggestions welcome
        Hide
        John George added a comment -

        @Mona - I have a patch that is almost ready, I will post it sometime today after I run somoe more tests and verify the fix.

        Show
        John George added a comment - @Mona - I have a patch that is almost ready, I will post it sometime today after I run somoe more tests and verify the fix.
        Hide
        Mona Chitnis added a comment -

        I'm hitting the problem 1

        testValidProxyUser(org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser): org.apache.hadoop.security.AccessControlException: Permission denied: user=u1, access=EXECUTE, inode="/home":<REPLACED_USER>:supergroup:drwxr-x---

        Is the patch for this JIRA underway?

        Show
        Mona Chitnis added a comment - I'm hitting the problem 1 testValidProxyUser(org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser): org.apache.hadoop.security.AccessControlException: Permission denied: user=u1, access=EXECUTE, inode="/home":<REPLACED_USER>:supergroup:drwxr-x--- Is the patch for this JIRA underway?
        Hide
        Siddharth Seth added a comment -

        The appjar would need to be read by the proxy (effective) user - so that it can be localized on the NM.

        4. Based on the johistory dir configuration - a single effective user may run, additional effective users per cluster could run into jobhistory dir permission issues.

        Show
        Siddharth Seth added a comment - The appjar would need to be read by the proxy (effective) user - so that it can be localized on the NM. 4. Based on the johistory dir configuration - a single effective user may run, additional effective users per cluster could run into jobhistory dir permission issues.
        Hide
        Mahadev konar added a comment -

        Sid,
        I am not sure I understand 3. Why does the proxy user need to read the AppJar? When referring to the Proxy user, it is the user who is doing the doAs right? I like to use effective user and real user (real user is the user who does the doAs imitating other users and effective user is the one being imitated).

        Show
        Mahadev konar added a comment - Sid, I am not sure I understand 3. Why does the proxy user need to read the AppJar? When referring to the Proxy user, it is the user who is doing the doAs right? I like to use effective user and real user (real user is the user who does the doAs imitating other users and effective user is the one being imitated).
        Hide
        Siddharth Seth added a comment -

        Tucu, after MR-3443, proxy users should work - at least the doAs part.
        There's several issues after that

        1. MiniMRYarnCluster - set the default app staging dir to the test Working dir. Ran into
        testValidProxyUser(org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser): Permission denied: user=u1, access=EXECUTE, inode="/home":<REPLACED_USER>:supergroup:drwxr-x---

        getJobConf needs -> conf.set(MRJobConfig.MR_AM_STAGING_DIR, "/tmp");

        2. The host list for the proxy user - needs to include the host IP, instead of just "localhost"

        3. After this, the proxy user cannot read the AppJar - since it was written out by the original user when creating the cluster (MiniMRClientClusterFactory).

        4. Another possible issue after this, would be creation of additional dirs for job history on hdfs.

        Show
        Siddharth Seth added a comment - Tucu, after MR-3443, proxy users should work - at least the doAs part. There's several issues after that 1. MiniMRYarnCluster - set the default app staging dir to the test Working dir. Ran into testValidProxyUser(org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser): Permission denied: user=u1, access=EXECUTE, inode="/home":<REPLACED_USER>:supergroup:drwxr-x--- getJobConf needs -> conf.set(MRJobConfig.MR_AM_STAGING_DIR, "/tmp"); 2. The host list for the proxy user - needs to include the host IP, instead of just "localhost" 3. After this, the proxy user cannot read the AppJar - since it was written out by the original user when creating the cluster (MiniMRClientClusterFactory). 4. Another possible issue after this, would be creation of additional dirs for job history on hdfs.
        Hide
        Alejandro Abdelnur added a comment -

        The attached testcase should pass if proxyuser settings work.

        Using the MR1 bridge API as I've written/verify the testcase with 0.20.20#.

        Show
        Alejandro Abdelnur added a comment - The attached testcase should pass if proxyuser settings work. Using the MR1 bridge API as I've written/verify the testcase with 0.20.20#.

          People

          • Assignee:
            John George
            Reporter:
            Alejandro Abdelnur
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development